Nov 03 19:39:04.592720 ip-10-0-128-24 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:39:05.022660 ip-10-0-128-24 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:39:05.022660 ip-10-0-128-24 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:39:05.022660 ip-10-0-128-24 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:39:05.022660 ip-10-0-128-24 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:39:05.029000 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.028872    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:39:05.029000 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.028995    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:39:05.029000 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.028999    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:39:05.029000 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029002    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:39:05.029000 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029004    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:39:05.029000 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029007    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029011    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029013    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029016    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029019    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029022    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029024    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029027    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029030    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029032    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029035    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029039    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029044    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029048    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029051    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029053    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029056    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029059    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029062    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:39:05.029250 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029064    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029068    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029071    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029074    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029077    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029085    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029089    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029092    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029094    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029097    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029100    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029103    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029105    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029108    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029111    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029114    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029117    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029121    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029124    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029127    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:39:05.029714 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029129    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029131    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029134    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029136    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029139    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029141    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029144    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029146    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029149    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029151    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029154    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029156    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029159    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029161    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029164    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029167    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029169    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029171    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029174    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029176    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:39:05.030173 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029179    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029181    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029184    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029187    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029189    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029192    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029194    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029197    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029200    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029202    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029205    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029207    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029210    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029212    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029214    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029217    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029219    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029221    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029224    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029227    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:39:05.030658 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029230    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029232    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029234    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029237    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029239    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029242    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029244    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029246    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029249    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029252    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029254    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029257    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029260    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029262    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029265    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029268    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029271    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029273    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029277    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:39:05.031131 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029280    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029283    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.029285    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030114    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030122    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030125    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030128    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030131    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030134    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030137    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030139    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030142    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030145    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030149    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030152    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030154    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030157    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030159    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030162    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030165    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:39:05.031602 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030168    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030170    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030173    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030176    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030180    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030184    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030187    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030190    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030193    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030195    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030198    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030200    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030203    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030206    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030208    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030211    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030214    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030216    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030219    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:39:05.032119 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030221    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030224    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030226    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030229    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030231    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030234    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030236    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030240    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030242    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030245    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030248    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030250    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030253    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030255    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030258    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030260    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030263    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030265    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030267    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:39:05.032586 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030271    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030276    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030279    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030281    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030284    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030286    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030290    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030293    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030295    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030298    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030300    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030303    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030305    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030308    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030310    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030313    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030315    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030317    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030320    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030323    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:39:05.033051 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030325    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030328    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030331    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030334    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030336    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030339    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030341    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030344    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030346    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030349    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030351    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030354    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030356    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030359    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030361    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030364    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030366    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030369    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030372    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030374    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:39:05.033545 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030377    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030379    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030381    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030384    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030387    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030389    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030392    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030394    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030397    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030399    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030403    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030896    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030903    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030906    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030909    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030912    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030915    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030917    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030920    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030922    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:39:05.034010 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030925    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030928    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030930    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030933    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030935    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030938    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030940    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030943    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030946    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030948    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030952    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030955    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030957    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030960    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030963    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030965    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030967    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030970    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030975    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030977    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:39:05.034490 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030980    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030983    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030985    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030987    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030990    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030992    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030995    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.030998    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031000    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031003    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031006    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031008    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031011    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031013    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031016    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031018    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031020    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031023    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031025    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031028    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:39:05.034962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031030    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031033    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031036    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031040    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031044    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031048    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031051    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031054    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031056    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031059    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031063    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031066    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031069    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031072    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031074    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031077    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031079    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031082    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031085    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:39:05.035421 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031088    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031090    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031092    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031095    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031097    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031100    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031102    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031105    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031107    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031111    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031114    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031117    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031121    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031123    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031126    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031128    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031131    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031133    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031136    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:39:05.035903 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031139    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031141    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031144    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031147    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031150    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031152    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031155    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031157    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031160    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031162    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031164    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031167    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031169    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031171    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031174    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031177    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031180    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031182    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.031186    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032336    2572 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:39:05.036388 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032348    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032356    2572 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032362    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032371    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032375    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032380    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032384    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032388    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032391    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032394    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032397    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032400    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032404    2572 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032407    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032409    2572 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032413    2572 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032416    2572 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032419    2572 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032423    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032426    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032442    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032446    2572 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032450    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:39:05.036867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032453    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032456    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032459    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032462    2572 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032467    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032470    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032474    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032476    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032481    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032484    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032487    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032490    2572 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032493    2572 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032496    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032501    2572 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032504    2572 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032507    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032510    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032512    2572 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032517    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032519    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032522    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032525    2572 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032528    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032531    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:39:05.037464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032534    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032536    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032539    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032542    2572 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032545    2572 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032549    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032552    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032555    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032558    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032560    2572 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032563    2572 flags.go:64] FLAG: --help="false"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032566    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032570    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032573    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032576    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032580    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032583    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032587    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032590    2572 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032592    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032595    2572 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032598    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032602    2572 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032604    2572 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:39:05.038057 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032607    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032610    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032613    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032616    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032619    2572 flags.go:64] FLAG: --lock-file=""
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032621    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032624    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032627    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032633    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032636    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032639    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032641    2572 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032644    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032648    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032650    2572 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032653    2572 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032657    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032660    2572 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032665    2572 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032667    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032670    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032674    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032676    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032680    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032683    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:39:05.038622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032688    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032696    2572 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032699    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032702    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032705    2572 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032708    2572 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032711    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032714    2572 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032717    2572 flags.go:64] FLAG: --port="10250"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032720    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032722    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0fe8f0d9aec6eeb9a"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032726    2572 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032728    2572 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032731    2572 flags.go:64] FLAG: --register-node="true"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032734    2572 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032738    2572 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032741    2572 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032744    2572 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032746    2572 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032750    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032753    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032757    2572 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032759    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032762    2572 flags.go:64] FLAG: --runonce="false"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032765    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032768    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032771    2572 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:39:05.039329 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032774    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032777    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032780    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032784    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032787    2572 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032791    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032794    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032797    2572 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032799    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032803    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032806    2572 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032809    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032814    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032817    2572 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032820    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032823    2572 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032826    2572 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032829    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032832    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032835    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032838    2572 flags.go:64] FLAG: --v="2"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032842    2572 flags.go:64] FLAG: --version="false"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032846    2572 flags.go:64] FLAG: --vmodule=""
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032850    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.032853    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:39:05.040004 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.032985    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.032988    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.032991    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.032994    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.032996    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.032999    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033001    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033004    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033011    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033013    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033016    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033018    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033024    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033027    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033029    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033032    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033034    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033037    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033039    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:39:05.040647 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033042    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033045    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033047    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033051    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033055    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033059    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033062    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033065    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033067    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033070    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033073    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033075    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033078    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033081    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033083    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033086    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033088    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033092    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033096    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:39:05.041092 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033099    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033101    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033105    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033107    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033109    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033113    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033117    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033119    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033122    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033124    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033127    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033129    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033132    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033134    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033137    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033139    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033141    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033144    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033146    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033149    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:39:05.041557 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033151    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033154    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033157    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033159    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033162    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033164    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033167    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033169    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033172    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033174    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033176    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033179    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033181    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033183    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033187    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033190    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033192    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033194    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033199    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033201    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:39:05.042029 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033204    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033207    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033210    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033212    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033215    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033217    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033220    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033222    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033224    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033227    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033229    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033232    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033234    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033237    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033239    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033241    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033244    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033246    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033248    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:39:05.042562 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033251    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033253    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033256    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033258    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033260    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033263    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033265    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033269    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033272    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.033298    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033333    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033338    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033343    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033346    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033349    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:39:05.043016 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033352    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033354    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033357    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033360    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033362    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033364    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033367    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033370    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033372    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033374    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033377    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033379    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033382    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033384    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033387    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033389    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033392    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033394    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033397    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:39:05.043368 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033399    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033401    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033404    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033406    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033408    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033412    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033415    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033417    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033420    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033422    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033425    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033441    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033444    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033447    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033450    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033452    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033455    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033457    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033460    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033462    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:39:05.043835 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033465    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033468    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033470    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033473    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033475    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033478    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033481    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033483    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033486    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033490    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033494    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033497    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033499    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033502    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033505    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033508    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033510    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033515    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033518    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033521    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:39:05.044311 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033524    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033526    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033530    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033534    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033539    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033542    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033544    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033547    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033550    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033552    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033555    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033558    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033560    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033563    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033565    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033568    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033570    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033573    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033575    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:39:05.044786 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033578    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033581    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033583    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033586    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033588    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033591    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033593    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033596    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033598    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033601    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033604    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033607    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033610    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033612    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033615    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033618    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033621    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033623    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033626    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033629    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:39:05.045223 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033632    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033634    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.033636    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.041126    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.041140    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.041157    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.041161    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041214    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041219    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041223    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041226    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041229    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041231    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041234    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041237    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041241    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041245    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041247    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041251    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:39:05.045726 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041253    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041256    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041258    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041261    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041264    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041266    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041269    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041272    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041274    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041277    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041279    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041282    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041284    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041287    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041289    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041292    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041294    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041298    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041300    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:39:05.046200 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041303    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041307    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041310    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041313    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041316    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041318    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041321    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041323    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041326    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041328    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041331    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041333    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041335    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041338    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041340    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041343    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041345    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041348    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041350    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:39:05.046656 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041353    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041355    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041358    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041360    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041362    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041365    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041368    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041370    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041372    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041375    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041377    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041380    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041383    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041385    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041388    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041390    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041393    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041395    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041397    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041399    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:39:05.047104 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041402    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041405    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041407    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041410    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041412    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041414    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041417    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041420    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041422    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041425    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041427    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041447    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041450    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041452    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041455    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041457    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041460    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041462    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041465    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:39:05.047595 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041467    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041469    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041472    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041474    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041476    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041479    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041482    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041485    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041487    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041490    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041492    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041495    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041497    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041500    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041502    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041505    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:39:05.048039 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041507    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.041535    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041692    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041696    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041699    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041703    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041707    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041710    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041714    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041717    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041720    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041723    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041726    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041728    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041730    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:39:05.048410 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041733    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041735    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041737    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041740    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041742    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041745    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041747    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041750    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041752    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041756    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041758    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041761    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041763    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041766    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041768    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041771    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041773    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041776    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041778    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:39:05.048783 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041780    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041783    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041785    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041787    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041790    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041793    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041796    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041799    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041801    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041804    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041806    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041808    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041811    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041813    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041815    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041818    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041820    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041822    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041825    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041827    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:39:05.049266 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041830    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041832    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041835    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041837    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041840    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041842    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041845    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041847    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041850    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041852    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041854    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041857    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041859    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041861    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041864    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041866    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041869    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041872    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041874    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041877    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:39:05.049738 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041879    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041881    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041884    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041886    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041888    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041891    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041893    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041896    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041898    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041901    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041903    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041906    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041908    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041911    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041913    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041915    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041918    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041920    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041923    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041925    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:39:05.050192 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041927    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041930    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041932    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041934    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041937    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041939    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041941    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041943    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041946    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041948    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041951    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041953    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041956    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.041958    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.041980    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042009    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:39:05.050772 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042013    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042016    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042019    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042022    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042024    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042027    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042029    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042032    2572 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042034    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042037    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042040    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042042    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042046    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042048    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042051    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042054    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042056    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042058    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042061    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:39:05.051176 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042063    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042065    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042068    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042071    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042073    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042075    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042078    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042080    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042082    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042085    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042089    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042092    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042094    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042097    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042099    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042101    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042104    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042107    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042109    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042112    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:39:05.051709 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042114    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042116    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042120    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042123    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042126    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042129    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042134    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042138    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042141    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042143    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042146    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042148    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042150    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042153    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042155    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042158    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042160    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042163    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:39:05.052175 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042165    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042168    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042170    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042172    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042176    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042178    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042181    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042184    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042186    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042188    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042191    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042193    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042196    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042198    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042201    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042203    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042205    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042208    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042210    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042213    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:39:05.052832 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042216    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042219    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042222    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042224    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042227    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042230    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042232    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042235    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042237    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042240    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042242    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042245    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042247    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042249    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042252    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042254    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042258    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042261    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042263    2572 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042266    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:39:05.053622 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042268    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042270    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042273    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042275    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042278    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042280    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042283    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:05.042286    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.042332    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.044757    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.045861    2572 server.go:1014] "Starting client certificate rotation"
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.045958    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.046898    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.048148    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:39:05.054180 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.051196    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:39:05.064678 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.064659    2572 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:39:05.068542 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.068527    2572 log.go:25] "Validated CRI v1 image API"
Nov 03 19:39:05.069338 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.069311    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:39:05.074511 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.074485    2572 fs.go:135] Filesystem UUIDs: map[63d116ec-017e-4fb1-b6e1-e15a9c9a4f56:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c769c5d2-a8eb-46cd-823d-42bdc89e9959:/dev/nvme0n1p4]
Nov 03 19:39:05.074511 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.074510    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 19:39:05.076076 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.076060    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:39:05.078738 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.078635    2572 manager.go:217] Machine: {Timestamp:2026-11-03 19:39:05.078164626 +0000 UTC m=+0.379779107 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098481 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2307ff6049715c009dd50d81b54870 SystemUUID:ec2307ff-6049-715c-009d-d50d81b54870 BootID:2f165dac-375f-40c2-b6b5-b64b115b9473 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:c4:98:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:c4:98:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:73:40:66:a4:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:39:05.078738 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.078729    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:39:05.078856 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.078797    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:39:05.082600 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.082575    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:39:05.082731 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.082601    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-24.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:39:05.082777 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.082736    2572 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:39:05.082777 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.082744    2572 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:39:05.082777 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.082756    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:39:05.082777 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.082769    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:39:05.084120 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.084110    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:39:05.088680 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.088668    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:39:05.088797 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.088787    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:39:05.088827 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.088820    2572 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:39:05.088853 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.088834    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:39:05.088879 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.088853    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:39:05.088879 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.088865    2572 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:39:05.088879 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.088877    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:39:05.089266 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.089255    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:39:05.091835 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.091820    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:39:05.099915 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.097295    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k6kpf"
Nov 03 19:39:05.101953 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.101932    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:39:05.102951 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.102932    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:39:05.103032 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.102976    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:39:05.103154 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.103142    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:39:05.103196 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.103167    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:39:05.103196 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.103181    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:39:05.103196 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.103189    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:39:05.103196 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.103196    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:39:05.103367 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.103206    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:39:05.104019 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.104007    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:39:05.104019 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.104019    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:39:05.104091 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.104027    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:39:05.104091 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.104032    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:39:05.104091 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.104041    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:39:05.104091 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.104051    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:39:05.104525 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.104505    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k6kpf"
Nov 03 19:39:05.105867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.105853    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:39:05.105867 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.105866    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:39:05.106419 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.106396    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 19:39:05.106482 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.106455    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-24.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 19:39:05.109884 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.109871    2572 server.go:1263] "Started kubelet"
Nov 03 19:39:05.110118 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.110058    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:39:05.110166 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.110159    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:39:05.113512 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.113296    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:39:05.113600 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.113557    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:39:05.113765 ip-10-0-128-24 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:39:05.114450 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.114411    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-24.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 19:39:05.117514 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.117494    2572 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:39:05.119481 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.119460    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 19:39:05.124165 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.124149    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:39:05.124814 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.124738    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:39:05.125197 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.125156    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:05.125699 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.125680    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:39:05.125699 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.125682    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:39:05.125859 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.125708    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:39:05.125859 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.125767    2572 factory.go:55] Registering systemd factory
Nov 03 19:39:05.125859 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.125782    2572 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:39:05.125859 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.125801    2572 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:39:05.125859 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.125811    2572 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:39:05.126036 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.126017    2572 factory.go:153] Registering CRI-O factory
Nov 03 19:39:05.126036 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.126031    2572 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:39:05.126129 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.126082    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:39:05.126129 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.126099    2572 factory.go:103] Registering Raw factory
Nov 03 19:39:05.126129 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.126113    2572 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:39:05.127240 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.127213    2572 manager.go:319] Starting recovery of all containers
Nov 03 19:39:05.128112 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.128091    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:39:05.130040 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.130019    2572 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-24.us-east-2.compute.internal\" not found" node="ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.139088 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.139071    2572 manager.go:324] Recovery completed
Nov 03 19:39:05.142783 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.142770    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:39:05.145294 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.145276    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:39:05.145372 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.145313    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:39:05.145372 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.145327    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:39:05.145953 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.145940    2572 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:39:05.146029 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.145953    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:39:05.146029 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.145973    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:39:05.146176 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.145956    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:39:05.146851 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.146835    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:39:05.146937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.146864    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:39:05.146937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.146874    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:39:05.148346 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.148326    2572 policy_none.go:50] "Start"
Nov 03 19:39:05.148346 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.148349    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:39:05.148558 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.148364    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:39:05.150646 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.150632    2572 policy_none.go:44] "Start"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.189026    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.193686    2572 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.194533    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.194552    2572 server.go:87] "Starting device plugin registration server"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.194786    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.194798    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.194873    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.194960    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.194973    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.195568    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.195641    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.197560    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.198800    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.198828    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.198850    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.198888    2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 19:39:05.203167 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.202221    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:39:05.295260 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.295186    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:39:05.296186 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.296170    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:39:05.296252 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.296202    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:39:05.296252 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.296213    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:39:05.296252 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.296238    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.299274 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.299259    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"]
Nov 03 19:39:05.299319 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.299311    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:39:05.300080 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.300057    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:39:05.300080 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.300083    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:39:05.300219 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.300094    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:39:05.302306 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.302294    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:39:05.302524 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.302496    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.302588 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.302528    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:39:05.302982 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.302968    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:39:05.303052 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.302992    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:39:05.303052 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.302996    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:39:05.303052 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.303008    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:39:05.303052 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.303017    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:39:05.303052 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.303027    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:39:05.305090 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.305072    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.305168 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.305100    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:39:05.305820 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.305801    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:39:05.305904 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.305832    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:39:05.305904 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.305846    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:39:05.307406 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.307392    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.307518 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.307412    2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-24.us-east-2.compute.internal\": node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:05.324374 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.324354    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:05.325806 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.325790    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found" node="ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.328508 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.328492    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found" node="ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.424631 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.424606    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:05.426834 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.426816    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.426898 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.426839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.426898 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.426858    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92c0a5455fd7c0c77f98e0c6b3176e58-config\") pod \"kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"92c0a5455fd7c0c77f98e0c6b3176e58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.525286 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.525244    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:05.527471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.527453    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.527524 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.527479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.527524 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.527500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92c0a5455fd7c0c77f98e0c6b3176e58-config\") pod \"kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"92c0a5455fd7c0c77f98e0c6b3176e58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.527595 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.527562    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92c0a5455fd7c0c77f98e0c6b3176e58-config\") pod \"kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"92c0a5455fd7c0c77f98e0c6b3176e58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.527595 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.527566    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.527666 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.527615    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.625943 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.625866    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:05.631196 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.631183    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.633378 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:05.633358    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:05.726827 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.726799    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:05.827228 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.827204    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:05.927757 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:05.927690    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:06.028258 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:06.028225    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:06.045508 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.045494    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 19:39:06.045644 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.045620    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:39:06.045686 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.045618    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:39:06.111016 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.110958    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:34:06 +0000 UTC" deadline="2028-04-04 03:11:33.537692154 +0000 UTC"
Nov 03 19:39:06.111016 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.111007    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12415h32m27.426687812s"
Nov 03 19:39:06.129210 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:06.129193    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:06.149678 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.149654    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:39:06.229751 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:06.229689    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:39:06.231899 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.231879    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:39:06.262147 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:06.262117    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c0a5455fd7c0c77f98e0c6b3176e58.slice/crio-6678bac546496281365a97a349a3cd41a1bc6a2055a4753a0ebb8f2dfe921f34 WatchSource:0}: Error finding container 6678bac546496281365a97a349a3cd41a1bc6a2055a4753a0ebb8f2dfe921f34: Status 404 returned error can't find the container with id 6678bac546496281365a97a349a3cd41a1bc6a2055a4753a0ebb8f2dfe921f34
Nov 03 19:39:06.262662 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:06.262642    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5aff924ba523805f662a53447cba5d.slice/crio-246af0f3d3dd0a114e9cdca6989f71416ae7e97c98ef154a92ddb038288644b6 WatchSource:0}: Error finding container 246af0f3d3dd0a114e9cdca6989f71416ae7e97c98ef154a92ddb038288644b6: Status 404 returned error can't find the container with id 246af0f3d3dd0a114e9cdca6989f71416ae7e97c98ef154a92ddb038288644b6
Nov 03 19:39:06.267757 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.267743    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:39:06.325870 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.325844    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:06.335854 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.335830    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:39:06.336938 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.336924    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:39:06.347187 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:06.347167    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:39:07.090581 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.090546    2572 apiserver.go:52] "Watching apiserver"
Nov 03 19:39:07.103998 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.103967    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:39:07.104900 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.104865    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-9sbm7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc","openshift-multus/multus-hqlmf","kube-system/konnectivity-agent-x8dsr","openshift-cluster-node-tuning-operator/tuned-txdfv","openshift-multus/multus-additional-cni-plugins-t8qlt","openshift-network-operator/iptables-alerter-4896h","openshift-ovn-kubernetes/ovnkube-node-ct5n8","kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal","openshift-image-registry/node-ca-hrqf7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-9bwkj","openshift-dns/node-resolver-zbfhf","openshift-multus/network-metrics-daemon-mr2m2"]
Nov 03 19:39:07.107549 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.107523    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:39:07.109910 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.109887    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:39:07.111803 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.111779    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:39:07.111923 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.111860    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:39:07.112022 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.112007    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:39:07.112074 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.112028    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zzp2g\""
Nov 03 19:39:07.113521 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.112936    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.113521 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.112988    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:39:07.113521 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.113071    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vmwsk\""
Nov 03 19:39:07.113521 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.113140    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:39:07.115490 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.115464    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:07.116199 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.116111    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:39:07.116199 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.116141    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:39:07.116199 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.116177    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:39:07.116199 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.116174    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:39:07.116445 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.116234    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rlc2f\""
Nov 03 19:39:07.118459 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.117946    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.118952 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.118934    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:39:07.119058 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.119039    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbmvx\""
Nov 03 19:39:07.119224 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.119203    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:39:07.120664 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.120642    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.123286 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.123029    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:07.123286 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.123098    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:07.124018 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.123958    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:39:07.124914 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.124277    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 19:39:07.125947 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.125552    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:07.125947 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.125613    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:07.128496 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.128190    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.130748 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.130728    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:39:07.134409 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.134391    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.134849 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.134829    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16525d8b-c267-41c1-8048-cec8f8dc381f-hosts-file\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:39:07.134997 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.134855    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-lib-modules\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.134997 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.134874    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpp59\" (UniqueName: \"kubernetes.io/projected/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-kube-api-access-kpp59\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.134997 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.134928    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.134997 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.134962    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-tuned\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.135297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135000    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-system-cni-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.135297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135028    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8jg\" (UniqueName: \"kubernetes.io/projected/576764d6-ba4a-416e-aac2-13e08c76680a-kube-api-access-2h8jg\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:39:07.135297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135052    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-bin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.135297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135075    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-hostroot\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.135297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135142    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/576764d6-ba4a-416e-aac2-13e08c76680a-iptables-alerter-script\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:39:07.135297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135169    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-netns\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.135297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135237    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-daemon-config\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.135297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135279    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16525d8b-c267-41c1-8048-cec8f8dc381f-tmp-dir\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:39:07.135828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135361    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.135828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135396    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:39:07.135828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135418    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.135828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135491    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-multus-certs\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.135828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqmc\" (UniqueName: \"kubernetes.io/projected/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-kube-api-access-5hqmc\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:07.135828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135569    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-conf\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.135828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135594    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-systemd\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.135828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135647    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.135828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135707    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-etc-selinux\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.135828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135758    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-system-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.135828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135800    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-registration-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135838    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-run\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135861    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-host\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135885    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-conf-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-etc-kubernetes\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135931    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-socket-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135965    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab539183-8f67-4c14-aef9-98b5924b6c33-agent-certs\") pod \"konnectivity-agent-x8dsr\" (UID: \"ab539183-8f67-4c14-aef9-98b5924b6c33\") " pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135983    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-modprobe-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.135998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-kubernetes\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136013    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/576764d6-ba4a-416e-aac2-13e08c76680a-host-slash\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136029    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136045    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136069    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-sys-fs\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136085    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab539183-8f67-4c14-aef9-98b5924b6c33-konnectivity-ca\") pod \"konnectivity-agent-x8dsr\" (UID: \"ab539183-8f67-4c14-aef9-98b5924b6c33\") " pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136128    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-tmp\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136144    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-cnibin\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136160    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-cnibin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.136392 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136176    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-dbus\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136192    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4znh8\" (UniqueName: \"kubernetes.io/projected/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kube-api-access-4znh8\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136207    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-sys\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136235    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136321    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-socket-dir-parent\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136383    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88nc2\" (UniqueName: \"kubernetes.io/projected/16525d8b-c267-41c1-8048-cec8f8dc381f-kube-api-access-88nc2\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136420    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-var-lib-kubelet\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136471    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136501    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-k8s-cni-cncf-io\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136609    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-kubelet\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136697    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-cni-binary-copy\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136728    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmh6\" (UniqueName: \"kubernetes.io/projected/a2ca8b68-6873-4723-ad8f-92876443f8fa-kube-api-access-qpmh6\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136757    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-device-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136784    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysconfig\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136833    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxts\" (UniqueName: \"kubernetes.io/projected/2fde71af-248d-498d-b358-4db5a52a8cd0-kube-api-access-qlxts\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.137382 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-multus\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.138287 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.136891    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-kubelet-config\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:07.138287 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.137137    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:07.138287 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.137184    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:07.159171 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.157488    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gcnmn"
Nov 03 19:39:07.166638 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.166610    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gcnmn"
Nov 03 19:39:07.205410 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.205363    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal" event={"ID":"92c0a5455fd7c0c77f98e0c6b3176e58","Type":"ContainerStarted","Data":"6678bac546496281365a97a349a3cd41a1bc6a2055a4753a0ebb8f2dfe921f34"}
Nov 03 19:39:07.206854 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.206812    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal" event={"ID":"dd5aff924ba523805f662a53447cba5d","Type":"ContainerStarted","Data":"246af0f3d3dd0a114e9cdca6989f71416ae7e97c98ef154a92ddb038288644b6"}
Nov 03 19:39:07.220498 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220470    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:39:07.220498 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220470    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:39:07.220669 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220651    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:39:07.220727 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220675    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:39:07.220727 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220714    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:39:07.220840 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220741    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:39:07.220840 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220763    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:39:07.220840 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220741    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rq9xl\""
Nov 03 19:39:07.220840 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220740    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zxqwg\""
Nov 03 19:39:07.220840 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220787    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:39:07.220840 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220781    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:39:07.220840 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.220838    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:39:07.221789 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.221770    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:39:07.221914 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.221902    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f2rb8\""
Nov 03 19:39:07.221990 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.221916    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:39:07.221990 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.221941    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:39:07.221990 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.221964    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:39:07.222196 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.222177    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d4wgq\""
Nov 03 19:39:07.222282 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.222200    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7xvgk\""
Nov 03 19:39:07.222282 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.222202    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:39:07.227111 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.227086    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:39:07.237428 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237394    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-system-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.237428 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237429    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-registration-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.237428 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237531    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-system-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.237428 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237597    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-registration-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.237810 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237655    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-netns\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.237810 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237677    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-etc-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.237810 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237694    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.237810 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237737    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-run\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.237810 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237788    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-host\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.238017 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237830    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-node-log\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.238017 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237853    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-host\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.238017 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237876    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-log-socket\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.238017 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237899    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-config\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.238017 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237923    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-conf-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.238017 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237929    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-run\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.238017 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.237970    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-etc-kubernetes\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.238017 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238012    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-conf-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238076    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-socket-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238098    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-etc-kubernetes\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238153    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab539183-8f67-4c14-aef9-98b5924b6c33-agent-certs\") pod \"konnectivity-agent-x8dsr\" (UID: \"ab539183-8f67-4c14-aef9-98b5924b6c33\") " pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238182    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-modprobe-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238210    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-kubernetes\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238250    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-socket-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238258    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-systemd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238302    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-kubernetes\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238312    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/576764d6-ba4a-416e-aac2-13e08c76680a-host-slash\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238341    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238369    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238344    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-modprobe-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238371    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/576764d6-ba4a-416e-aac2-13e08c76680a-host-slash\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:39:07.238420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238395    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-sys-fs\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238421    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab539183-8f67-4c14-aef9-98b5924b6c33-konnectivity-ca\") pod \"konnectivity-agent-x8dsr\" (UID: \"ab539183-8f67-4c14-aef9-98b5924b6c33\") " pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.238471    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.238472    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238490    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-tmp\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238520    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-var-lib-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.238531    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret podName:e2aab3d5-357d-470a-a27d-bab9d570a73d nodeName:}" failed. No retries permitted until 2026-11-03 19:39:07.738510218 +0000 UTC m=+3.040124703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret") pod "global-pull-secret-syncer-9sbm7" (UID: "e2aab3d5-357d-470a-a27d-bab9d570a73d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238540    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-sys-fs\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.238553    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs podName:eb3bbe34-cf5f-4d4f-88b9-600903ab0961 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:07.738542274 +0000 UTC m=+3.040156746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs") pod "network-metrics-daemon-mr2m2" (UID: "eb3bbe34-cf5f-4d4f-88b9-600903ab0961") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238587    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-cnibin\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238617    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-cnibin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238622    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-dbus\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238669    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znh8\" (UniqueName: \"kubernetes.io/projected/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kube-api-access-4znh8\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238694    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-sys\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238717    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-cnibin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238721    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.239119 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238773    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-socket-dir-parent\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238784    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-dbus\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88nc2\" (UniqueName: \"kubernetes.io/projected/16525d8b-c267-41c1-8048-cec8f8dc381f-kube-api-access-88nc2\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238777    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-sys\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-var-lib-kubelet\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238853    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-socket-dir-parent\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-script-lib\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238894    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238947    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0034d15b-8b67-4e49-8abd-b834b164af4f-host\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238977    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-k8s-cni-cncf-io\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238986    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-var-lib-kubelet\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.238807    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-cnibin\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239017    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239023    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-kubelet\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239040    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239068    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-kubelet\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239078    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-k8s-cni-cncf-io\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.240060 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239081    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-slash\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239160    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab539183-8f67-4c14-aef9-98b5924b6c33-konnectivity-ca\") pod \"konnectivity-agent-x8dsr\" (UID: \"ab539183-8f67-4c14-aef9-98b5924b6c33\") " pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239187    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239212    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grjf\" (UniqueName: \"kubernetes.io/projected/f6bc998d-830b-4bed-9536-4e9a6424afe3-kube-api-access-7grjf\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239275    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-cni-binary-copy\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239412    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmh6\" (UniqueName: \"kubernetes.io/projected/a2ca8b68-6873-4723-ad8f-92876443f8fa-kube-api-access-qpmh6\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239480    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-device-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239631    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-device-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239670    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysconfig\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239701    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxts\" (UniqueName: \"kubernetes.io/projected/2fde71af-248d-498d-b358-4db5a52a8cd0-kube-api-access-qlxts\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239726    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239752    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0034d15b-8b67-4e49-8abd-b834b164af4f-serviceca\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239771    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovn-node-metrics-cert\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239789    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239791    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-cni-binary-copy\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239807    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-env-overrides\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.240937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239835    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-multus\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239850    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysconfig\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239854    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-kubelet-config\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239892    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-kubelet-config\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239893    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16525d8b-c267-41c1-8048-cec8f8dc381f-hosts-file\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239934    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16525d8b-c267-41c1-8048-cec8f8dc381f-hosts-file\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239938    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-lib-modules\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.239971    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpp59\" (UniqueName: \"kubernetes.io/projected/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-kube-api-access-kpp59\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240025    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240063    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-lib-modules\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240077    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-tuned\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240139    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-system-cni-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240140    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-multus\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8jg\" (UniqueName: \"kubernetes.io/projected/576764d6-ba4a-416e-aac2-13e08c76680a-kube-api-access-2h8jg\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240221    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-bin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240257    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-hostroot\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240286    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-bin\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.241794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240316    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-netd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240343    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/576764d6-ba4a-416e-aac2-13e08c76680a-iptables-alerter-script\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240421    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-bin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240464    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-system-cni-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240509    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-hostroot\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240563    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-netns\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240608    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-daemon-config\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240635    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-netns\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240669    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16525d8b-c267-41c1-8048-cec8f8dc381f-tmp-dir\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240687    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slxrz\" (UniqueName: \"kubernetes.io/projected/0034d15b-8b67-4e49-8abd-b834b164af4f-kube-api-access-slxrz\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240702    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240729    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-systemd-units\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240790    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240829    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240927    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-multus-certs\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240946    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqmc\" (UniqueName: \"kubernetes.io/projected/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-kube-api-access-5hqmc\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:07.242471 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.240963    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-conf\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.241022    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-systemd\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.241052    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.241079    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-ovn\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.241106    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-etc-selinux\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.241155    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-kubelet\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.241335    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-conf\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.241587    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-systemd\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.241697    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.241702    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-multus-certs\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.241694    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-etc-selinux\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.241775    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/576764d6-ba4a-416e-aac2-13e08c76680a-iptables-alerter-script\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:39:07.243042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.243017    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.243416 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.243084    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-daemon-config\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.243416 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.243230    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16525d8b-c267-41c1-8048-cec8f8dc381f-tmp-dir\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:39:07.243416 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.243272    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-tmp\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.243416 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.243318    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-tuned\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.243566 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.243531    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab539183-8f67-4c14-aef9-98b5924b6c33-agent-certs\") pod \"konnectivity-agent-x8dsr\" (UID: \"ab539183-8f67-4c14-aef9-98b5924b6c33\") " pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:07.252954 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.252924    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.256706 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.256678    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znh8\" (UniqueName: \"kubernetes.io/projected/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kube-api-access-4znh8\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.257464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.257432    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88nc2\" (UniqueName: \"kubernetes.io/projected/16525d8b-c267-41c1-8048-cec8f8dc381f-kube-api-access-88nc2\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:39:07.258512 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.258490    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqmc\" (UniqueName: \"kubernetes.io/projected/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-kube-api-access-5hqmc\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:07.259111 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.259084    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8jg\" (UniqueName: \"kubernetes.io/projected/576764d6-ba4a-416e-aac2-13e08c76680a-kube-api-access-2h8jg\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:39:07.259729 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.259706    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxts\" (UniqueName: \"kubernetes.io/projected/2fde71af-248d-498d-b358-4db5a52a8cd0-kube-api-access-qlxts\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.260475 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.260443    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpp59\" (UniqueName: \"kubernetes.io/projected/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-kube-api-access-kpp59\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.262001 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.261933    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmh6\" (UniqueName: \"kubernetes.io/projected/a2ca8b68-6873-4723-ad8f-92876443f8fa-kube-api-access-qpmh6\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.342031 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.341992    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-slash\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342228 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342039    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grjf\" (UniqueName: \"kubernetes.io/projected/f6bc998d-830b-4bed-9536-4e9a6424afe3-kube-api-access-7grjf\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342228 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342128    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-slash\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342364 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342253    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:07.342364 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0034d15b-8b67-4e49-8abd-b834b164af4f-serviceca\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:39:07.342364 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342312    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovn-node-metrics-cert\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342554 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342416    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342554 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342471    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-env-overrides\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342554 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342506    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-bin\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342554 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342546    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-netd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342756 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342555    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-bin\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342756 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342600    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342756 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342586    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slxrz\" (UniqueName: \"kubernetes.io/projected/0034d15b-8b67-4e49-8abd-b834b164af4f-kube-api-access-slxrz\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:39:07.342756 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342663    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-netd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342756 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342736    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-systemd-units\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342971 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342768    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342971 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342793    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-ovn\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342971 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342823    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-kubelet\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342971 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342830    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-systemd-units\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342971 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342852    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-netns\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342971 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342865    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342971 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342933    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-etc-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.342971 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342960    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0034d15b-8b67-4e49-8abd-b834b164af4f-serviceca\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:39:07.343242 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342963    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343242 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.342996    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343242 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343025    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-node-log\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343242 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343036    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-kubelet\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343242 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343057    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-env-overrides\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343242 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343070    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-ovn\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343242 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343078    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-netns\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343242 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343102    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-log-socket\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343242 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343114    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-node-log\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343242 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343154    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-etc-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343242 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343182    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-config\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343708 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343315    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-log-socket\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343708 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343389    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-systemd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343708 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343469    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-var-lib-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343708 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343493    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-systemd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343708 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343509    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-script-lib\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343708 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343534    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0034d15b-8b67-4e49-8abd-b834b164af4f-host\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:39:07.343708 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343551    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-var-lib-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343708 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343601    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0034d15b-8b67-4e49-8abd-b834b164af4f-host\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:39:07.343937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343717    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-config\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.343937 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.343911    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-script-lib\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.345393 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.345365    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovn-node-metrics-cert\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.348348 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.348326    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:39:07.348348 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.348351    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:39:07.348563 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.348367    2572 projected.go:196] Error preparing data for projected volume kube-api-access-69f2f for pod openshift-network-diagnostics/network-check-target-9bwkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:07.348563 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.348431    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f podName:d70bada2-ec8a-40db-80d9-6455a9a675a3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:07.848411409 +0000 UTC m=+3.150025904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-69f2f" (UniqueName: "kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f") pod "network-check-target-9bwkj" (UID: "d70bada2-ec8a-40db-80d9-6455a9a675a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:07.351214 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.351193    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grjf\" (UniqueName: \"kubernetes.io/projected/f6bc998d-830b-4bed-9536-4e9a6424afe3-kube-api-access-7grjf\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.351635 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.351611    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slxrz\" (UniqueName: \"kubernetes.io/projected/0034d15b-8b67-4e49-8abd-b834b164af4f-kube-api-access-slxrz\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:39:07.425368 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.425019    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:39:07.434256 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.434225    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:39:07.439474 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:07.439425    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576764d6_ba4a_416e_aac2_13e08c76680a.slice/crio-cc72b4ca04bc904a8d9e4c670bc021fe46d1623d6e1bdf0fb2b65e77b6736fdb WatchSource:0}: Error finding container cc72b4ca04bc904a8d9e4c670bc021fe46d1623d6e1bdf0fb2b65e77b6736fdb: Status 404 returned error can't find the container with id cc72b4ca04bc904a8d9e4c670bc021fe46d1623d6e1bdf0fb2b65e77b6736fdb
Nov 03 19:39:07.445705 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.445681    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hqlmf"
Nov 03 19:39:07.446898 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:07.446869    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16525d8b_c267_41c1_8048_cec8f8dc381f.slice/crio-aa90567a31bf21c64a5651c62d660a07942d209f34ee252ff8a214d4e389da57 WatchSource:0}: Error finding container aa90567a31bf21c64a5651c62d660a07942d209f34ee252ff8a214d4e389da57: Status 404 returned error can't find the container with id aa90567a31bf21c64a5651c62d660a07942d209f34ee252ff8a214d4e389da57
Nov 03 19:39:07.451716 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.451690    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:07.458337 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:07.458309    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ca8b68_6873_4723_ad8f_92876443f8fa.slice/crio-21752a7a0138e2305b6882d40ed02003ee0660c837bbfa7a2c48283d107a22be WatchSource:0}: Error finding container 21752a7a0138e2305b6882d40ed02003ee0660c837bbfa7a2c48283d107a22be: Status 404 returned error can't find the container with id 21752a7a0138e2305b6882d40ed02003ee0660c837bbfa7a2c48283d107a22be
Nov 03 19:39:07.463072 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.463049    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:39:07.463708 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:07.463679    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab539183_8f67_4c14_aef9_98b5924b6c33.slice/crio-b587f04ed3984d763957dd90913c8bc07819228a05c9e8749b16df29c9299ff3 WatchSource:0}: Error finding container b587f04ed3984d763957dd90913c8bc07819228a05c9e8749b16df29c9299ff3: Status 404 returned error can't find the container with id b587f04ed3984d763957dd90913c8bc07819228a05c9e8749b16df29c9299ff3
Nov 03 19:39:07.473744 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:07.473718    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8ed709_4fdf_40b1_99c6_63e0d2a20e25.slice/crio-5952f4e02bb62e4043333ad6f3f0c0d5b797045ae8b7e9d1e03d97ab91d05bad WatchSource:0}: Error finding container 5952f4e02bb62e4043333ad6f3f0c0d5b797045ae8b7e9d1e03d97ab91d05bad: Status 404 returned error can't find the container with id 5952f4e02bb62e4043333ad6f3f0c0d5b797045ae8b7e9d1e03d97ab91d05bad
Nov 03 19:39:07.489352 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.489326    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:39:07.492475 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.492439    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:39:07.494664 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.494623    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:39:07.500807 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:07.500787    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fde71af_248d_498d_b358_4db5a52a8cd0.slice/crio-dd674b248c6165f5d22a07c5a067c2db45322c3d1a86dcf72083d284a8d16775 WatchSource:0}: Error finding container dd674b248c6165f5d22a07c5a067c2db45322c3d1a86dcf72083d284a8d16775: Status 404 returned error can't find the container with id dd674b248c6165f5d22a07c5a067c2db45322c3d1a86dcf72083d284a8d16775
Nov 03 19:39:07.504151 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:07.504128    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0034d15b_8b67_4e49_8abd_b834b164af4f.slice/crio-8ae954350a2552cfef72be13db2b78f3546858a6c9221b01e1a27213b3c791bd WatchSource:0}: Error finding container 8ae954350a2552cfef72be13db2b78f3546858a6c9221b01e1a27213b3c791bd: Status 404 returned error can't find the container with id 8ae954350a2552cfef72be13db2b78f3546858a6c9221b01e1a27213b3c791bd
Nov 03 19:39:07.507962 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:07.507929    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c1c0da_1ec1_4a75_9017_430bcedcc19b.slice/crio-cf1d6b248740da36f5e293f6636bbf18e1c3da50d6144b5b437a9c702063cc51 WatchSource:0}: Error finding container cf1d6b248740da36f5e293f6636bbf18e1c3da50d6144b5b437a9c702063cc51: Status 404 returned error can't find the container with id cf1d6b248740da36f5e293f6636bbf18e1c3da50d6144b5b437a9c702063cc51
Nov 03 19:39:07.513561 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.513540    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:07.517756 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.517736    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:39:07.524929 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:07.524905    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bc998d_830b_4bed_9536_4e9a6424afe3.slice/crio-48f72ab27ab404e9d978dfaf7e3d68a71d5db54d62bddf7f4cf8743da250db58 WatchSource:0}: Error finding container 48f72ab27ab404e9d978dfaf7e3d68a71d5db54d62bddf7f4cf8743da250db58: Status 404 returned error can't find the container with id 48f72ab27ab404e9d978dfaf7e3d68a71d5db54d62bddf7f4cf8743da250db58
Nov 03 19:39:07.533990 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.533942    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:39:07.746692 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.746007    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:07.746692 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.746051    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:07.746692 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.746204    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:07.746692 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.746268    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret podName:e2aab3d5-357d-470a-a27d-bab9d570a73d nodeName:}" failed. No retries permitted until 2026-11-03 19:39:08.746248619 +0000 UTC m=+4.047863106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret") pod "global-pull-secret-syncer-9sbm7" (UID: "e2aab3d5-357d-470a-a27d-bab9d570a73d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:07.746692 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.746654    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:07.747020 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.746716    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs podName:eb3bbe34-cf5f-4d4f-88b9-600903ab0961 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:08.746698743 +0000 UTC m=+4.048313212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs") pod "network-metrics-daemon-mr2m2" (UID: "eb3bbe34-cf5f-4d4f-88b9-600903ab0961") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:07.948051 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:07.947647    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:07.948051 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.947805    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:39:07.948051 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.947825    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:39:07.948051 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.947842    2572 projected.go:196] Error preparing data for projected volume kube-api-access-69f2f for pod openshift-network-diagnostics/network-check-target-9bwkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:07.948051 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:07.947903    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f podName:d70bada2-ec8a-40db-80d9-6455a9a675a3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:08.947884224 +0000 UTC m=+4.249498697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-69f2f" (UniqueName: "kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f") pod "network-check-target-9bwkj" (UID: "d70bada2-ec8a-40db-80d9-6455a9a675a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:08.170800 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.170433    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:34:08 +0000 UTC" deadline="2028-06-28 09:44:06.747895573 +0000 UTC"
Nov 03 19:39:08.170800 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.170742    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14462h4m58.577159937s"
Nov 03 19:39:08.199938 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.199351    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:08.199938 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:08.199542    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:08.222069 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.222028    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal" event={"ID":"92c0a5455fd7c0c77f98e0c6b3176e58","Type":"ContainerStarted","Data":"8c91e656d32f823c7b8102c1099bead51d014fc5f3ca94ac37cc540a1947e1e8"}
Nov 03 19:39:08.233371 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.233254    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" event={"ID":"f6bc998d-830b-4bed-9536-4e9a6424afe3","Type":"ContainerStarted","Data":"48f72ab27ab404e9d978dfaf7e3d68a71d5db54d62bddf7f4cf8743da250db58"}
Nov 03 19:39:08.239548 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.238927    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc" event={"ID":"81c1c0da-1ec1-4a75-9017-430bcedcc19b","Type":"ContainerStarted","Data":"cf1d6b248740da36f5e293f6636bbf18e1c3da50d6144b5b437a9c702063cc51"}
Nov 03 19:39:08.239548 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.239086    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal" podStartSLOduration=1.239067315 podStartE2EDuration="1.239067315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:08.238861844 +0000 UTC m=+3.540476338" watchObservedRunningTime="2026-11-03 19:39:08.239067315 +0000 UTC m=+3.540681818"
Nov 03 19:39:08.244341 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.244275    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hrqf7" event={"ID":"0034d15b-8b67-4e49-8abd-b834b164af4f","Type":"ContainerStarted","Data":"8ae954350a2552cfef72be13db2b78f3546858a6c9221b01e1a27213b3c791bd"}
Nov 03 19:39:08.249567 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.249490    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x8dsr" event={"ID":"ab539183-8f67-4c14-aef9-98b5924b6c33","Type":"ContainerStarted","Data":"b587f04ed3984d763957dd90913c8bc07819228a05c9e8749b16df29c9299ff3"}
Nov 03 19:39:08.252801 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.252721    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hqlmf" event={"ID":"a2ca8b68-6873-4723-ad8f-92876443f8fa","Type":"ContainerStarted","Data":"21752a7a0138e2305b6882d40ed02003ee0660c837bbfa7a2c48283d107a22be"}
Nov 03 19:39:08.254569 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.254506    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8qlt" event={"ID":"2fde71af-248d-498d-b358-4db5a52a8cd0","Type":"ContainerStarted","Data":"dd674b248c6165f5d22a07c5a067c2db45322c3d1a86dcf72083d284a8d16775"}
Nov 03 19:39:08.261161 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.261136    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-txdfv" event={"ID":"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25","Type":"ContainerStarted","Data":"5952f4e02bb62e4043333ad6f3f0c0d5b797045ae8b7e9d1e03d97ab91d05bad"}
Nov 03 19:39:08.263476 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.263403    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zbfhf" event={"ID":"16525d8b-c267-41c1-8048-cec8f8dc381f","Type":"ContainerStarted","Data":"aa90567a31bf21c64a5651c62d660a07942d209f34ee252ff8a214d4e389da57"}
Nov 03 19:39:08.265205 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.265180    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4896h" event={"ID":"576764d6-ba4a-416e-aac2-13e08c76680a","Type":"ContainerStarted","Data":"cc72b4ca04bc904a8d9e4c670bc021fe46d1623d6e1bdf0fb2b65e77b6736fdb"}
Nov 03 19:39:08.755187 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.754321    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:08.755187 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.754373    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:08.755187 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:08.754581    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:08.755187 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:08.754653    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret podName:e2aab3d5-357d-470a-a27d-bab9d570a73d nodeName:}" failed. No retries permitted until 2026-11-03 19:39:10.754632605 +0000 UTC m=+6.056247078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret") pod "global-pull-secret-syncer-9sbm7" (UID: "e2aab3d5-357d-470a-a27d-bab9d570a73d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:08.755187 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:08.755015    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:08.755187 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:08.755069    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs podName:eb3bbe34-cf5f-4d4f-88b9-600903ab0961 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:10.755052869 +0000 UTC m=+6.056667355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs") pod "network-metrics-daemon-mr2m2" (UID: "eb3bbe34-cf5f-4d4f-88b9-600903ab0961") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:08.956418 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:08.956376    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:08.956666 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:08.956586    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:39:08.956666 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:08.956607    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:39:08.956666 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:08.956621    2572 projected.go:196] Error preparing data for projected volume kube-api-access-69f2f for pod openshift-network-diagnostics/network-check-target-9bwkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:08.956832 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:08.956717    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f podName:d70bada2-ec8a-40db-80d9-6455a9a675a3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:10.956674302 +0000 UTC m=+6.258288786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-69f2f" (UniqueName: "kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f") pod "network-check-target-9bwkj" (UID: "d70bada2-ec8a-40db-80d9-6455a9a675a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:09.171252 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:09.171202    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:34:08 +0000 UTC" deadline="2028-06-09 12:21:58.568253606 +0000 UTC"
Nov 03 19:39:09.171252 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:09.171249    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14008h42m49.397009333s"
Nov 03 19:39:09.199766 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:09.199732    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:09.199969 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:09.199899    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:09.200532 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:09.200335    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:09.200532 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:09.200474    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:09.270571 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:09.270393    2572 generic.go:356] "Generic (PLEG): container finished" podID="dd5aff924ba523805f662a53447cba5d" containerID="a8cdb9cc8396d06ff4385b752eb04121cf98a4edd91fa856c4813a4cb940d70d" exitCode=0
Nov 03 19:39:09.270761 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:09.270586    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal" event={"ID":"dd5aff924ba523805f662a53447cba5d","Type":"ContainerDied","Data":"a8cdb9cc8396d06ff4385b752eb04121cf98a4edd91fa856c4813a4cb940d70d"}
Nov 03 19:39:10.199560 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:10.199525    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:10.200100 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:10.199681    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:10.288782 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:10.288034    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal" event={"ID":"dd5aff924ba523805f662a53447cba5d","Type":"ContainerStarted","Data":"e7df215cddae7f6bb2d8bc2d724df0585817e0ab684301bc12a7291ef9dfae58"}
Nov 03 19:39:10.304158 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:10.304098    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal" podStartSLOduration=3.304080592 podStartE2EDuration="3.304080592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:10.303938528 +0000 UTC m=+5.605553026" watchObservedRunningTime="2026-11-03 19:39:10.304080592 +0000 UTC m=+5.605695087"
Nov 03 19:39:10.772693 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:10.771963    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:10.772693 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:10.772009    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:10.772693 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:10.772200    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:10.772693 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:10.772202    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:10.772693 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:10.772273    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret podName:e2aab3d5-357d-470a-a27d-bab9d570a73d nodeName:}" failed. No retries permitted until 2026-11-03 19:39:14.772253881 +0000 UTC m=+10.073868367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret") pod "global-pull-secret-syncer-9sbm7" (UID: "e2aab3d5-357d-470a-a27d-bab9d570a73d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:10.772693 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:10.772305    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs podName:eb3bbe34-cf5f-4d4f-88b9-600903ab0961 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:14.772283925 +0000 UTC m=+10.073898410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs") pod "network-metrics-daemon-mr2m2" (UID: "eb3bbe34-cf5f-4d4f-88b9-600903ab0961") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:10.973764 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:10.973723    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:10.973940 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:10.973919    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:39:10.973999 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:10.973946    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:39:10.973999 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:10.973961    2572 projected.go:196] Error preparing data for projected volume kube-api-access-69f2f for pod openshift-network-diagnostics/network-check-target-9bwkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:10.974121 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:10.974020    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f podName:d70bada2-ec8a-40db-80d9-6455a9a675a3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:14.974001343 +0000 UTC m=+10.275615818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-69f2f" (UniqueName: "kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f") pod "network-check-target-9bwkj" (UID: "d70bada2-ec8a-40db-80d9-6455a9a675a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:11.201710 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:11.201613    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:11.202206 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:11.201896    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:11.209853 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:11.209810    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:11.209998 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:11.209972    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:11.295259 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:11.295215    2572 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="57f96087f383ab4daa9314e99aca14641a157f8c49b8cb1b3510a8756cd142c7" exitCode=0
Nov 03 19:39:11.295611 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:11.295579    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8qlt" event={"ID":"2fde71af-248d-498d-b358-4db5a52a8cd0","Type":"ContainerDied","Data":"57f96087f383ab4daa9314e99aca14641a157f8c49b8cb1b3510a8756cd142c7"}
Nov 03 19:39:12.199599 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:12.199544    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:12.199863 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:12.199703    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:12.302687 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:12.302652    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x8dsr" event={"ID":"ab539183-8f67-4c14-aef9-98b5924b6c33","Type":"ContainerStarted","Data":"88755db61ea544faab776d081608c0cd0c1a13f46cc134f702cc85a0cbdee574"}
Nov 03 19:39:13.202232 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:13.201583    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:13.202232 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:13.201723    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:13.202232 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:13.202070    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:13.202232 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:13.202172    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:14.199510 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:14.199411    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:14.200073 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:14.199572    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:14.310483 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:14.310436    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc" event={"ID":"81c1c0da-1ec1-4a75-9017-430bcedcc19b","Type":"ContainerStarted","Data":"10f75e471c3dbcc8e12edaf6e05e0442fafc08eba560d4f44a4b5a47aa93df86"}
Nov 03 19:39:14.313607 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:14.313577    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hrqf7" event={"ID":"0034d15b-8b67-4e49-8abd-b834b164af4f","Type":"ContainerStarted","Data":"7abe3cedaa7e3fdaf33a9365133f829d0f5d09f5d9d6683dcbb2440ff72047f6"}
Nov 03 19:39:14.330096 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:14.330050    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x8dsr" podStartSLOduration=4.085670702 podStartE2EDuration="8.330036335s" totalImagesPullingTime="4.244365633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:12.322547589 +0000 UTC m=+7.624162087" watchObservedRunningTime="2026-11-03 19:39:14.330036335 +0000 UTC m=+9.631650828"
Nov 03 19:39:14.812140 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:14.812100    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:14.812292 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:14.812157    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:14.812349 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:14.812298    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:14.812384 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:14.812365    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret podName:e2aab3d5-357d-470a-a27d-bab9d570a73d nodeName:}" failed. No retries permitted until 2026-11-03 19:39:22.812345913 +0000 UTC m=+18.113960385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret") pod "global-pull-secret-syncer-9sbm7" (UID: "e2aab3d5-357d-470a-a27d-bab9d570a73d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:14.812839 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:14.812814    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:14.812956 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:14.812879    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs podName:eb3bbe34-cf5f-4d4f-88b9-600903ab0961 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:22.812861391 +0000 UTC m=+18.114475865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs") pod "network-metrics-daemon-mr2m2" (UID: "eb3bbe34-cf5f-4d4f-88b9-600903ab0961") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:15.014038 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:15.013994    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:15.020844 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:15.020804    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:39:15.020844 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:15.020850    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:39:15.021065 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:15.020877    2572 projected.go:196] Error preparing data for projected volume kube-api-access-69f2f for pod openshift-network-diagnostics/network-check-target-9bwkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:15.021065 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:15.020969    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f podName:d70bada2-ec8a-40db-80d9-6455a9a675a3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:23.020938983 +0000 UTC m=+18.322553457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-69f2f" (UniqueName: "kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f") pod "network-check-target-9bwkj" (UID: "d70bada2-ec8a-40db-80d9-6455a9a675a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:15.203013 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:15.202977    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:15.203402 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:15.203122    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:15.205654 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:15.203575    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:15.205654 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:15.203673    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:15.767128 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:15.767089    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:39:15.917749 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:15.917714    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:15.919236 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:15.919179    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:15.942222 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:15.942166    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hrqf7" podStartSLOduration=4.304260328 podStartE2EDuration="9.942149925s" totalImagesPullingTime="5.637889597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:14.330720338 +0000 UTC m=+9.632334832" watchObservedRunningTime="2026-11-03 19:39:15.942149925 +0000 UTC m=+11.243764415"
Nov 03 19:39:16.199992 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:16.199959    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:16.200179 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:16.200105    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:16.210210 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:16.210130    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:39:15.767106693Z","UUID":"1106bd55-5933-414c-9200-749050916293","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:39:16.212032 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:16.212006    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:39:16.212126 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:16.212040    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:39:16.323707 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:16.323667    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc" event={"ID":"81c1c0da-1ec1-4a75-9017-430bcedcc19b","Type":"ContainerStarted","Data":"2e71f5a06fd3b8730701b31a9d2cc871c3e74cacb639595be38433c0dc4e399f"}
Nov 03 19:39:16.325495 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:16.325444    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hqlmf" event={"ID":"a2ca8b68-6873-4723-ad8f-92876443f8fa","Type":"ContainerStarted","Data":"efb902166a26ca81c4a92978f5e731febd1e41c3acfef3aac5a017c37c28166a"}
Nov 03 19:39:16.327723 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:16.327697    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zbfhf" event={"ID":"16525d8b-c267-41c1-8048-cec8f8dc381f","Type":"ContainerStarted","Data":"a313dff1add74d1d3e239bf5bad83df47b0b19483a523ae3448426ddb3da4f16"}
Nov 03 19:39:16.344660 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:16.344610    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hqlmf" podStartSLOduration=1.992592057 podStartE2EDuration="10.344592294s" totalImagesPullingTime="8.352000237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:16.344047342 +0000 UTC m=+11.645661835" watchObservedRunningTime="2026-11-03 19:39:16.344592294 +0000 UTC m=+11.646206789"
Nov 03 19:39:16.360224 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:16.360168    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zbfhf" podStartSLOduration=1.955479162 podStartE2EDuration="10.360150163s" totalImagesPullingTime="8.404671001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:16.359268351 +0000 UTC m=+11.660882842" watchObservedRunningTime="2026-11-03 19:39:16.360150163 +0000 UTC m=+11.661764658"
Nov 03 19:39:17.201797 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:17.201491    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:17.201971 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:17.201921    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:17.201971 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:17.201542    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:17.202080 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:17.202030    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:17.333357 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:17.333259    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc" event={"ID":"81c1c0da-1ec1-4a75-9017-430bcedcc19b","Type":"ContainerStarted","Data":"c2e38ef23a41e6edd9e33f7dd316b8b8e80e94deb1f920a40a437b58f1fae41a"}
Nov 03 19:39:17.336195 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:17.335646    2572 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="ba0b89d577fa63e9ff7db151f4c420aefdf254ce886b2cdf49ce90588c7c8f92" exitCode=0
Nov 03 19:39:17.336195 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:17.335772    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8qlt" event={"ID":"2fde71af-248d-498d-b358-4db5a52a8cd0","Type":"ContainerDied","Data":"ba0b89d577fa63e9ff7db151f4c420aefdf254ce886b2cdf49ce90588c7c8f92"}
Nov 03 19:39:17.352660 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:17.352608    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc" podStartSLOduration=2.329121755 podStartE2EDuration="11.352590399s" totalImagesPullingTime="9.023468644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:17.352560115 +0000 UTC m=+12.654174609" watchObservedRunningTime="2026-11-03 19:39:17.352590399 +0000 UTC m=+12.654204917"
Nov 03 19:39:18.199662 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:18.199627    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:18.199802 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:18.199761    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:18.340369 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:18.340285    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4896h" event={"ID":"576764d6-ba4a-416e-aac2-13e08c76680a","Type":"ContainerStarted","Data":"1579a9c316d2ebf828d26fc153fa8ec17877e9777311f3b72e8c963be8e32f1b"}
Nov 03 19:39:19.199713 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:19.199678    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:19.199713 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:19.199718    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:19.199949 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:19.199811    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:19.200005 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:19.199958    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:19.344773 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:19.344734    2572 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="cb3878b75c9491860cebbd7701f6561c580c1ed809d2dc5004945fdc92d4701b" exitCode=0
Nov 03 19:39:19.345521 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:19.344828    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8qlt" event={"ID":"2fde71af-248d-498d-b358-4db5a52a8cd0","Type":"ContainerDied","Data":"cb3878b75c9491860cebbd7701f6561c580c1ed809d2dc5004945fdc92d4701b"}
Nov 03 19:39:19.346827 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:19.346807    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-txdfv" event={"ID":"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25","Type":"ContainerStarted","Data":"31b4b1640f4991de4a9c1d92eb478d18cd88a99bd35ef2d03df06e026bd616da"}
Nov 03 19:39:19.364701 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:19.364650    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4896h" podStartSLOduration=4.527505279 podStartE2EDuration="13.364632396s" totalImagesPullingTime="8.837127117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:18.355637048 +0000 UTC m=+13.657251544" watchObservedRunningTime="2026-11-03 19:39:19.364632396 +0000 UTC m=+14.666246881"
Nov 03 19:39:19.378750 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:19.378700    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-txdfv" podStartSLOduration=2.399556347 podStartE2EDuration="13.37868915s" totalImagesPullingTime="10.979132803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:19.378074708 +0000 UTC m=+14.679689198" watchObservedRunningTime="2026-11-03 19:39:19.37868915 +0000 UTC m=+14.680303640"
Nov 03 19:39:20.199596 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:20.199552    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:20.199754 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:20.199698    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:20.350893 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:20.350862    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8qlt" event={"ID":"2fde71af-248d-498d-b358-4db5a52a8cd0","Type":"ContainerStarted","Data":"26f95f8c7c2d0697449a44a4d927a4c71f356b74e024ff3ccb3e67efb4abcaa5"}
Nov 03 19:39:21.199313 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:21.199277    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:21.199527 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:21.199459    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:21.199527 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:21.199497    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:21.199641 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:21.199603    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:21.355950 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:21.355916    2572 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="26f95f8c7c2d0697449a44a4d927a4c71f356b74e024ff3ccb3e67efb4abcaa5" exitCode=0
Nov 03 19:39:21.356428 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:21.355981    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8qlt" event={"ID":"2fde71af-248d-498d-b358-4db5a52a8cd0","Type":"ContainerDied","Data":"26f95f8c7c2d0697449a44a4d927a4c71f356b74e024ff3ccb3e67efb4abcaa5"}
Nov 03 19:39:22.200161 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:22.199970    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:22.200400 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:22.200249    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:22.881759 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:22.881716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:22.882232 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:22.881770    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:22.882232 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:22.881893    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:22.882232 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:22.881953    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret podName:e2aab3d5-357d-470a-a27d-bab9d570a73d nodeName:}" failed. No retries permitted until 2026-11-03 19:39:38.881936235 +0000 UTC m=+34.183550704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret") pod "global-pull-secret-syncer-9sbm7" (UID: "e2aab3d5-357d-470a-a27d-bab9d570a73d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:22.882232 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:22.882011    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:22.882232 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:22.882038    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs podName:eb3bbe34-cf5f-4d4f-88b9-600903ab0961 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:38.882028657 +0000 UTC m=+34.183643125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs") pod "network-metrics-daemon-mr2m2" (UID: "eb3bbe34-cf5f-4d4f-88b9-600903ab0961") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:23.083754 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:23.083722    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:23.083917 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:23.083900    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:39:23.083971 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:23.083924    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:39:23.083971 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:23.083937    2572 projected.go:196] Error preparing data for projected volume kube-api-access-69f2f for pod openshift-network-diagnostics/network-check-target-9bwkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:23.084059 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:23.083992    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f podName:d70bada2-ec8a-40db-80d9-6455a9a675a3 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:39.083974486 +0000 UTC m=+34.385588954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-69f2f" (UniqueName: "kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f") pod "network-check-target-9bwkj" (UID: "d70bada2-ec8a-40db-80d9-6455a9a675a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:23.203297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:23.202970    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:23.203297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:23.202993    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:23.203427 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:23.203318    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:23.203427 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:23.203335    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:23.363414 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:23.363379    2572 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="b91debb8fa0b8f21db59d794bb9847bc9d07316edd68d60b22711705b739a3e4" exitCode=0
Nov 03 19:39:23.363571 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:23.363474    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8qlt" event={"ID":"2fde71af-248d-498d-b358-4db5a52a8cd0","Type":"ContainerDied","Data":"b91debb8fa0b8f21db59d794bb9847bc9d07316edd68d60b22711705b739a3e4"}
Nov 03 19:39:23.366876 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:23.366849    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" event={"ID":"f6bc998d-830b-4bed-9536-4e9a6424afe3","Type":"ContainerStarted","Data":"25cdae7f172f13e5bf22b482f03459b0da5fb3ea39818ec1dd52a5ca5359216a"}
Nov 03 19:39:23.366982 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:23.366935    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" event={"ID":"f6bc998d-830b-4bed-9536-4e9a6424afe3","Type":"ContainerStarted","Data":"3966f29377e0426a596e6724456bce30b7d707ac0c2337379587d93aada6d880"}
Nov 03 19:39:23.367047 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:23.367030    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" event={"ID":"f6bc998d-830b-4bed-9536-4e9a6424afe3","Type":"ContainerStarted","Data":"b37a3f8a4b881123e7c6c64f396e2b16469b8c80313bd088f6eeda766da9af87"}
Nov 03 19:39:23.367103 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:23.367089    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" event={"ID":"f6bc998d-830b-4bed-9536-4e9a6424afe3","Type":"ContainerStarted","Data":"e9f2b57e920b9bf63abb515f2a105ff34e13e51265fc395ce41b79ef94c44389"}
Nov 03 19:39:23.367159 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:23.367146    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" event={"ID":"f6bc998d-830b-4bed-9536-4e9a6424afe3","Type":"ContainerStarted","Data":"27fd8fb391292629a190f93aff6d7161ddcf3d50bd9c90c6f6f2713d9b803f27"}
Nov 03 19:39:23.367213 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:23.367200    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" event={"ID":"f6bc998d-830b-4bed-9536-4e9a6424afe3","Type":"ContainerStarted","Data":"ad8c37ac17498efd729d13854ecd2545e79c94377dee3dd5c7ff18d0cdc707f2"}
Nov 03 19:39:24.199931 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:24.199899    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:24.200321 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:24.200014    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:24.204054 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:24.204030    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:24.204161 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:24.204148    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:39:24.204619 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:24.204599    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:39:24.371960 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:24.371884    2572 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="8e440fc30202cf5b1daedc53f0f497b1447bc3f03e91956bb14a365e2a1a5e22" exitCode=0
Nov 03 19:39:24.372122 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:24.371962    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8qlt" event={"ID":"2fde71af-248d-498d-b358-4db5a52a8cd0","Type":"ContainerDied","Data":"8e440fc30202cf5b1daedc53f0f497b1447bc3f03e91956bb14a365e2a1a5e22"}
Nov 03 19:39:25.199957 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:25.199919    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:25.200472 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:25.200016    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:25.200472 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:25.200067    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:25.200472 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:25.200147    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:25.377349 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:25.377259    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t8qlt" event={"ID":"2fde71af-248d-498d-b358-4db5a52a8cd0","Type":"ContainerStarted","Data":"3ad3ba5227a281b1627c84b241a60165565a35fbe70304de30ff8551ef70b7dc"}
Nov 03 19:39:25.412381 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:25.412309    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t8qlt" podStartSLOduration=7.228727388 podStartE2EDuration="19.412295062s" totalImagesPullingTime="12.183567674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:25.411856254 +0000 UTC m=+20.713470745" watchObservedRunningTime="2026-11-03 19:39:25.412295062 +0000 UTC m=+20.713909594"
Nov 03 19:39:26.199641 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:26.199608    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:26.199787 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:26.199716    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:26.382910 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:26.382871    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" event={"ID":"f6bc998d-830b-4bed-9536-4e9a6424afe3","Type":"ContainerStarted","Data":"4006cf42fd8bcc63257ba284d58ff10e50e5978d304b42790db6871b7f1da525"}
Nov 03 19:39:27.202595 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:27.202562    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:27.202595 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:27.202597    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:27.202790 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:27.202677    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:27.202823 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:27.202791    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:28.199903 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:28.199875    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:28.200502 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:28.199979    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:28.390724 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:28.390686    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" event={"ID":"f6bc998d-830b-4bed-9536-4e9a6424afe3","Type":"ContainerStarted","Data":"2b3f3e259fb4b56355db4463f77865f8dda1c7cc3eec778fc230c612359f1a06"}
Nov 03 19:39:28.390917 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:28.390891    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:28.407223 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:28.407201    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:28.436634 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:28.436595    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" podStartSLOduration=7.015118978 podStartE2EDuration="22.436582211s" totalImagesPullingTime="15.421463233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:28.436095203 +0000 UTC m=+23.737709693" watchObservedRunningTime="2026-11-03 19:39:28.436582211 +0000 UTC m=+23.738196725"
Nov 03 19:39:29.204152 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:29.204121    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:29.204152 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:29.204156    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:29.204617 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:29.204238    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:29.204617 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:29.204356    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:29.393328 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:29.393302    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:39:29.393742 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:29.393722    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:29.409448 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:29.409409    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:30.199880 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:30.199850    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:30.200054 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:30.199955    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:30.216252 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:30.216231    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zbfhf_16525d8b-c267-41c1-8048-cec8f8dc381f/dns-node-resolver/0.log"
Nov 03 19:39:30.395603 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:30.395576    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:39:30.798260 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:30.798234    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hrqf7_0034d15b-8b67-4e49-8abd-b834b164af4f/node-ca/0.log"
Nov 03 19:39:31.200058 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:31.200027    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:31.200219 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:31.200070    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:31.200219 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:31.200154    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:31.200280 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:31.200259    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:31.397584 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:31.397557    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:39:32.199512 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:32.199474    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:32.199685 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:32.199622    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:33.199653 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:33.199613    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:33.200343 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:33.199718    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:33.200343 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:33.199796    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:33.200343 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:33.199883    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:34.199533 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:34.199497    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:34.199716 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:34.199621    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:35.200448 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:35.200397    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:35.200918 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:35.200552    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:35.200918 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:35.200598    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:35.200918 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:35.200684    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:36.199953 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:36.199919    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:36.200119 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:36.200026    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:36.214900 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:36.214876    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:39:36.215277 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:36.215103    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:39:36.225655 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:36.225620    2572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" podUID="f6bc998d-830b-4bed-9536-4e9a6424afe3" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 19:39:36.234188 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:36.234162    2572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8" podUID="f6bc998d-830b-4bed-9536-4e9a6424afe3" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 19:39:37.199728 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:37.199693    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:37.199953 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:37.199795    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:37.199953 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:37.199699    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:37.199953 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:37.199854    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:38.199756 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:38.199720    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:38.200204 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:38.199829    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:38.889835 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:38.889800    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:38.889835 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:38.889833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:38.890037 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:38.889935    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:38.890037 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:38.889949    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:38.890037 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:38.889984    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret podName:e2aab3d5-357d-470a-a27d-bab9d570a73d nodeName:}" failed. No retries permitted until 2026-11-03 19:40:10.889971275 +0000 UTC m=+66.191585744 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret") pod "global-pull-secret-syncer-9sbm7" (UID: "e2aab3d5-357d-470a-a27d-bab9d570a73d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:38.890037 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:38.890003    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs podName:eb3bbe34-cf5f-4d4f-88b9-600903ab0961 nodeName:}" failed. No retries permitted until 2026-11-03 19:40:10.889989651 +0000 UTC m=+66.191604120 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs") pod "network-metrics-daemon-mr2m2" (UID: "eb3bbe34-cf5f-4d4f-88b9-600903ab0961") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:39.091864 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:39.091831    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:39.092014 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:39.091994    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:39:39.092056 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:39.092014    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:39:39.092056 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:39.092026    2572 projected.go:196] Error preparing data for projected volume kube-api-access-69f2f for pod openshift-network-diagnostics/network-check-target-9bwkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:39.092118 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:39.092081    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f podName:d70bada2-ec8a-40db-80d9-6455a9a675a3 nodeName:}" failed. No retries permitted until 2026-11-03 19:40:11.092067779 +0000 UTC m=+66.393682248 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-69f2f" (UniqueName: "kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f") pod "network-check-target-9bwkj" (UID: "d70bada2-ec8a-40db-80d9-6455a9a675a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:39.199097 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:39.199030    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:39.199235 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:39.199030    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:39.199235 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:39.199142    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:39.199235 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:39.199198    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:40.200010 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:40.199978    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:40.200396 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:40.200080    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:41.199293 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:41.199258    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:41.199293 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:41.199290    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:41.199546 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:41.199378    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:41.199546 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:41.199521    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:42.199771 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:42.199740    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:42.200143 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:42.199837    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:43.200123 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:43.200083    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:43.200123 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:43.200118    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:43.200684 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:43.200214    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:43.200684 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:43.200328    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:44.199042 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:44.199002    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:44.199234 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:44.199134    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:45.200230 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:45.200195    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:45.200674 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:45.200279    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:45.200674 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:45.200352    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:45.200674 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:45.200454    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:46.199554 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:46.199517    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:46.199714 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:46.199655    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:47.203297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:47.203259    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:47.203297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:47.203282    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:47.203712 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:47.203399    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:47.203712 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:47.203520    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:47.392518 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:47.392486    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mr2m2"]
Nov 03 19:39:47.392676 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:47.392663    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:47.392814 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:47.392789    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:47.393145 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:47.393125    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9sbm7"]
Nov 03 19:39:47.393634 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:47.393610    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9bwkj"]
Nov 03 19:39:47.432409 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:47.432389    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:47.432512 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:47.432419    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:47.432623 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:47.432519    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:47.432703 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:47.432637    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:49.202881 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:49.202708    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:49.203359 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:49.202761    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:49.203359 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:49.202770    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:49.203359 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:49.202944    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:49.203359 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:49.203017    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:49.203359 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:49.203128    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:51.202153 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.202116    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:51.202153 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.202156    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:51.202847 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.202224    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:51.202847 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:51.202242    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mr2m2" podUID="eb3bbe34-cf5f-4d4f-88b9-600903ab0961"
Nov 03 19:39:51.202847 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:51.202293    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bwkj" podUID="d70bada2-ec8a-40db-80d9-6455a9a675a3"
Nov 03 19:39:51.202847 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:51.202344    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9sbm7" podUID="e2aab3d5-357d-470a-a27d-bab9d570a73d"
Nov 03 19:39:51.436184 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.436156    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeReady"
Nov 03 19:39:51.436337 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.436265    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 19:39:51.483245 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.483218    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zgwk6"]
Nov 03 19:39:51.505670 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.505640    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-btkpj"]
Nov 03 19:39:51.506467 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.506171    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.514352 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.514333    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tlzjx\""
Nov 03 19:39:51.514428 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.514412    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:39:51.515782 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.515764    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:39:51.521659 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.521639    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zgwk6"]
Nov 03 19:39:51.521749 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.521676    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-btkpj"]
Nov 03 19:39:51.521749 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.521706    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-62g8g"]
Nov 03 19:39:51.521824 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.521773    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.535876 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.535859    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62g8g"]
Nov 03 19:39:51.535970 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.535950    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:39:51.610408 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.610375    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mvxmh\""
Nov 03 19:39:51.610408 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.610394    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:39:51.610636 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.610463    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hnqg\""
Nov 03 19:39:51.610636 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.610624    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:39:51.617245 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.617229    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:39:51.617357 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.617344    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:39:51.617420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.617391    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:39:51.617420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.617393    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:39:51.617420 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.617400    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:39:51.684765 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.684728    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvvg\" (UniqueName: \"kubernetes.io/projected/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-kube-api-access-rqvvg\") pod \"ingress-canary-62g8g\" (UID: \"2224bfc1-cc82-4e90-b723-e629d5f0f3f5\") " pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:39:51.684765 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.684764    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-tmp-dir\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.684958 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.684858    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.684958 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.684888    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-metrics-tls\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.684958 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.684930    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8bedff4e-9e2b-4582-bafe-29360cfa83ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.685058 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.684992    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-cert\") pod \"ingress-canary-62g8g\" (UID: \"2224bfc1-cc82-4e90-b723-e629d5f0f3f5\") " pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:39:51.685058 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.685017    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bedff4e-9e2b-4582-bafe-29360cfa83ae-data-volume\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.685126 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.685067    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jnr\" (UniqueName: \"kubernetes.io/projected/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-api-access-v8jnr\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.685126 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.685104    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bphw8\" (UniqueName: \"kubernetes.io/projected/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-kube-api-access-bphw8\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.685192 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.685128    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8bedff4e-9e2b-4582-bafe-29360cfa83ae-crio-socket\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.685192 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.685143    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-config-volume\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.786126 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786041    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8bedff4e-9e2b-4582-bafe-29360cfa83ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.786126 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786083    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-cert\") pod \"ingress-canary-62g8g\" (UID: \"2224bfc1-cc82-4e90-b723-e629d5f0f3f5\") " pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:39:51.786126 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786103    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bedff4e-9e2b-4582-bafe-29360cfa83ae-data-volume\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.786126 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786121    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jnr\" (UniqueName: \"kubernetes.io/projected/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-api-access-v8jnr\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.786462 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786139    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bphw8\" (UniqueName: \"kubernetes.io/projected/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-kube-api-access-bphw8\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.786462 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786166    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8bedff4e-9e2b-4582-bafe-29360cfa83ae-crio-socket\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.786462 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786188    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-config-volume\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.786462 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786217    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvvg\" (UniqueName: \"kubernetes.io/projected/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-kube-api-access-rqvvg\") pod \"ingress-canary-62g8g\" (UID: \"2224bfc1-cc82-4e90-b723-e629d5f0f3f5\") " pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:39:51.786462 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786244    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-tmp-dir\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.786462 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786293    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.786462 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786314    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-metrics-tls\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.786766 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786583    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bedff4e-9e2b-4582-bafe-29360cfa83ae-data-volume\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.786766 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786652    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8bedff4e-9e2b-4582-bafe-29360cfa83ae-crio-socket\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.786960 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.786939    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-tmp-dir\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.787116 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.787094    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.787180 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.787158    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-config-volume\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.790221 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.790194    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8bedff4e-9e2b-4582-bafe-29360cfa83ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.790319 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.790263    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-metrics-tls\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.790319 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.790267    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-cert\") pod \"ingress-canary-62g8g\" (UID: \"2224bfc1-cc82-4e90-b723-e629d5f0f3f5\") " pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:39:51.793818 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.793789    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphw8\" (UniqueName: \"kubernetes.io/projected/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-kube-api-access-bphw8\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.793911 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.793861    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jnr\" (UniqueName: \"kubernetes.io/projected/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-api-access-v8jnr\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.794202 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.794180    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvvg\" (UniqueName: \"kubernetes.io/projected/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-kube-api-access-rqvvg\") pod \"ingress-canary-62g8g\" (UID: \"2224bfc1-cc82-4e90-b723-e629d5f0f3f5\") " pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:39:51.842870 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.842847    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:51.892482 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.892459    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:39:51.906153 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:51.906122    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:39:52.207233 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.207200    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-btkpj"]
Nov 03 19:39:52.208870 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.208850    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zgwk6"]
Nov 03 19:39:52.219575 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.219555    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62g8g"]
Nov 03 19:39:52.222034 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:52.221948    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2224bfc1_cc82_4e90_b723_e629d5f0f3f5.slice/crio-0b6f018189450cd956b8b25de13d8b2f5c6c0f236fd7835b69315c4840131a8a WatchSource:0}: Error finding container 0b6f018189450cd956b8b25de13d8b2f5c6c0f236fd7835b69315c4840131a8a: Status 404 returned error can't find the container with id 0b6f018189450cd956b8b25de13d8b2f5c6c0f236fd7835b69315c4840131a8a
Nov 03 19:39:52.443851 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.443641    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgwk6" event={"ID":"8668d2b7-8bf0-4786-b96f-0ba9ab45472c","Type":"ContainerStarted","Data":"55ed9d78bcef62604e945b7540a963036fb859a46533d9904b9c429d5d0b2048"}
Nov 03 19:39:52.444730 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.444703    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62g8g" event={"ID":"2224bfc1-cc82-4e90-b723-e629d5f0f3f5","Type":"ContainerStarted","Data":"0b6f018189450cd956b8b25de13d8b2f5c6c0f236fd7835b69315c4840131a8a"}
Nov 03 19:39:52.445968 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.445950    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btkpj" event={"ID":"8bedff4e-9e2b-4582-bafe-29360cfa83ae","Type":"ContainerStarted","Data":"9a7065bc485c890eb8659740d6e037910e8d9b23ed90e938f18e4bee0b331dce"}
Nov 03 19:39:52.446031 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.445988    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btkpj" event={"ID":"8bedff4e-9e2b-4582-bafe-29360cfa83ae","Type":"ContainerStarted","Data":"39b2397af6c1565c258e80f46815d70fe4d820256d5b76c46112a390b842ade8"}
Nov 03 19:39:52.887752 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.887709    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"]
Nov 03 19:39:52.899714 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.899674    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"]
Nov 03 19:39:52.901078 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.901050    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:52.912770 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.912744    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 19:39:52.912902 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.912875    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vgtlv\""
Nov 03 19:39:52.913035 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.913019    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 19:39:52.916657 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.916570    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:39:52.916800 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.916748    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:39:52.916865 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.916854    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:39:52.922228 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.922130    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"]
Nov 03 19:39:52.922228 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.922181    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"]
Nov 03 19:39:52.922399 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.922276    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:52.925821 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.925798    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zftbb"]
Nov 03 19:39:52.937532 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.936117    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nbbcm\""
Nov 03 19:39:52.937532 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.936157    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 19:39:52.937532 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.936234    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 19:39:52.949581 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:52.949555    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.014026 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.013997    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:39:53.014218 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.014094    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g7dvr\""
Nov 03 19:39:53.014218 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.014132    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:39:53.017190 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.017169    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:39:53.017312 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.017269    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 19:39:53.096892 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.096859    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rp8\" (UniqueName: \"kubernetes.io/projected/5d178b87-95d6-4985-bf0f-f0448dd784d7-kube-api-access-k6rp8\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.097074 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.096913    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.097074 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.096947    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-wtmp\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.097074 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.096970    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.097074 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.096991    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.097074 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097008    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-metrics-client-ca\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.097074 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097031    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-tls\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.097363 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097090    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.097363 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097118    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.097363 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097176    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/edcd5516-336e-415d-b587-c887774d8a63-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.097363 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097213    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.097363 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097241    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd8vt\" (UniqueName: \"kubernetes.io/projected/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-kube-api-access-jd8vt\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.097363 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097276    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw8cx\" (UniqueName: \"kubernetes.io/projected/edcd5516-336e-415d-b587-c887774d8a63-kube-api-access-fw8cx\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.097363 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097300    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-root\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.097363 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097336    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.097363 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097356    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-textfile\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.097766 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097385    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.097766 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097415    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.097766 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.097641    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-sys\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198331    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198382    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198409    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-sys\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198465    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rp8\" (UniqueName: \"kubernetes.io/projected/5d178b87-95d6-4985-bf0f-f0448dd784d7-kube-api-access-k6rp8\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198497    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198526    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-wtmp\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198550    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198580    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198608    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-metrics-client-ca\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198634    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-tls\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198753    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/edcd5516-336e-415d-b587-c887774d8a63-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198780    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.198806    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd8vt\" (UniqueName: \"kubernetes.io/projected/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-kube-api-access-jd8vt\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.204509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.201269    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-sys\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.205747 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.202777    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw8cx\" (UniqueName: \"kubernetes.io/projected/edcd5516-336e-415d-b587-c887774d8a63-kube-api-access-fw8cx\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.205747 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.202811    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-root\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.205747 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.202851    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.205747 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.202878    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-textfile\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.205747 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.203140    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-textfile\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.205747 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.203140    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.209296 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.209269    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/edcd5516-336e-415d-b587-c887774d8a63-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.210663 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.210642    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.210777 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.210763    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-wtmp\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.211328 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.211301    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.211949 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.211928    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.212904 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.212388    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.212904 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.212512    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-root\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.214153 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.214128    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.214649 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.214626    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-metrics-client-ca\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.215600 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.215574    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-tls\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.215678 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.215581    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.215678 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.215581    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.216024 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.216007    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.219722 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.219704    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:39:53.220046 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.220029    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:39:53.220489 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.220423    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:39:53.222803 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.222778    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rp8\" (UniqueName: \"kubernetes.io/projected/5d178b87-95d6-4985-bf0f-f0448dd784d7-kube-api-access-k6rp8\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.223178 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.223157    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw8cx\" (UniqueName: \"kubernetes.io/projected/edcd5516-336e-415d-b587-c887774d8a63-kube-api-access-fw8cx\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.225783 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.225759    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd8vt\" (UniqueName: \"kubernetes.io/projected/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-kube-api-access-jd8vt\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.233814 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.230520    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:39:53.233814 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.230627    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:39:53.233814 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.231220    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-drbgz\""
Nov 03 19:39:53.233814 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.231310    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zcfm6\""
Nov 03 19:39:53.233814 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.231320    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:39:53.233814 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.231391    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:39:53.233814 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.231486    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5rk8\""
Nov 03 19:39:53.256334 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.256108    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:39:53.320150 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.320121    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:39:53.366344 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.366305    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:39:53.387042 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:53.386938    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d178b87_95d6_4985_bf0f_f0448dd784d7.slice/crio-27b11c5c6597d0250624a1141508bb56fd612ede51505a8053d7445197e92a6b WatchSource:0}: Error finding container 27b11c5c6597d0250624a1141508bb56fd612ede51505a8053d7445197e92a6b: Status 404 returned error can't find the container with id 27b11c5c6597d0250624a1141508bb56fd612ede51505a8053d7445197e92a6b
Nov 03 19:39:53.452115 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.452072    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zftbb" event={"ID":"5d178b87-95d6-4985-bf0f-f0448dd784d7","Type":"ContainerStarted","Data":"27b11c5c6597d0250624a1141508bb56fd612ede51505a8053d7445197e92a6b"}
Nov 03 19:39:53.659643 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.659610    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"]
Nov 03 19:39:53.663503 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:53.663469    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcd5516_336e_415d_b587_c887774d8a63.slice/crio-ad6d6c671a0cc6d8d687fcf5cf5f30dd3f1661857d4d9bd1488991d60f18c718 WatchSource:0}: Error finding container ad6d6c671a0cc6d8d687fcf5cf5f30dd3f1661857d4d9bd1488991d60f18c718: Status 404 returned error can't find the container with id ad6d6c671a0cc6d8d687fcf5cf5f30dd3f1661857d4d9bd1488991d60f18c718
Nov 03 19:39:53.680962 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.680887    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"]
Nov 03 19:39:53.683467 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:53.683418    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3eb2db_f7a4_4fd0_8fdd_e84c11d822b1.slice/crio-e7f7e72c047d1f63fa136e8bdf58d06b70d33d94913245861076f11e7eeadedb WatchSource:0}: Error finding container e7f7e72c047d1f63fa136e8bdf58d06b70d33d94913245861076f11e7eeadedb: Status 404 returned error can't find the container with id e7f7e72c047d1f63fa136e8bdf58d06b70d33d94913245861076f11e7eeadedb
Nov 03 19:39:53.982095 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.980303    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:39:53.985676 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.984293    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:53.994711 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.994313    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 19:39:53.994711 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.994495    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 19:39:53.994711 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.994571    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 19:39:53.994711 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.994609    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 19:39:53.994924 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.994744    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 19:39:53.994924 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.994811    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 19:39:53.994924 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.994872    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 19:39:53.995059 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.994976    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 19:39:53.995125 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.995100    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kcnh9\""
Nov 03 19:39:53.995264 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.995242    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 19:39:53.998765 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:53.998745    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:39:54.012302 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012274    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-web-config\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012418 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012306    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012418 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012333    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012418 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012627 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012511    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012627 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012544    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012627 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012569    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qzh\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-kube-api-access-s5qzh\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012627 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-out\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012821 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012639    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012821 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012706    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012821 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012729    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-volume\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012821 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012757    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.012821 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.012810    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.113833 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.113797    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114016 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.113842    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-volume\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114016 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.113871    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114016 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.113899    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114016 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.113924    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-web-config\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114016 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.113951    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114016 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.113979    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114016 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.114009    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114350 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.114069    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114350 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.114095    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114350 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.114122    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qzh\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-kube-api-access-s5qzh\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114350 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.114165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-out\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114350 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.114194    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.114860 ip-10-0-128-24 kubenswrapper[2572]: E1103 19:39:54.114837    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle podName:1fd1ca4f-1929-48df-8769-b2e469863d7a nodeName:}" failed. No retries permitted until 2026-11-03 19:39:54.614812119 +0000 UTC m=+49.916426592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a") : configmap references non-existent config key: ca-bundle.crt
Nov 03 19:39:54.116984 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.116919    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.117787 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.117703    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.119584 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.119553    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.122843 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.121829    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-volume\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.122843 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.121869    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.122843 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.121908    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-out\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.122843 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.122374    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.122843 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.122799    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.123778 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.123727    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.124012 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.123987    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.124497 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.124476    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-web-config\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.126027 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.126004    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qzh\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-kube-api-access-s5qzh\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.465840 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.465800    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btkpj" event={"ID":"8bedff4e-9e2b-4582-bafe-29360cfa83ae","Type":"ContainerStarted","Data":"7a2cc4ae752222968d4c021a7b4a822002c0f9c8e20b54178a931c86524f3d10"}
Nov 03 19:39:54.468713 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.468591    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d" event={"ID":"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1","Type":"ContainerStarted","Data":"7991f206457aaa855cb70c8f3db62de6cb317c3c0064d9945f121e6c9744580f"}
Nov 03 19:39:54.468713 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.468646    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d" event={"ID":"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1","Type":"ContainerStarted","Data":"05fe75634e72da80f6df411759e9c66a99f8b15194d526b424798105c1fa929c"}
Nov 03 19:39:54.468713 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.468673    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d" event={"ID":"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1","Type":"ContainerStarted","Data":"e7f7e72c047d1f63fa136e8bdf58d06b70d33d94913245861076f11e7eeadedb"}
Nov 03 19:39:54.471401 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.471337    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7" event={"ID":"edcd5516-336e-415d-b587-c887774d8a63","Type":"ContainerStarted","Data":"ad6d6c671a0cc6d8d687fcf5cf5f30dd3f1661857d4d9bd1488991d60f18c718"}
Nov 03 19:39:54.619139 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.619113    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.620637 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.620068    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:54.916745 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:54.916483    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:39:55.233138 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:55.233017    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:39:55.235589 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:55.235540    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd1ca4f_1929_48df_8769_b2e469863d7a.slice/crio-a5ae888670d147d2333357f46f183055d310272c3896cbf5baed60d208c08d6f WatchSource:0}: Error finding container a5ae888670d147d2333357f46f183055d310272c3896cbf5baed60d208c08d6f: Status 404 returned error can't find the container with id a5ae888670d147d2333357f46f183055d310272c3896cbf5baed60d208c08d6f
Nov 03 19:39:55.481725 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:55.481687    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgwk6" event={"ID":"8668d2b7-8bf0-4786-b96f-0ba9ab45472c","Type":"ContainerStarted","Data":"06d71221258f3d83841297c71a7da0a14c7a4e56b7f13a0ba4ea622a2d110a0c"}
Nov 03 19:39:55.482219 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:55.481785    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgwk6" event={"ID":"8668d2b7-8bf0-4786-b96f-0ba9ab45472c","Type":"ContainerStarted","Data":"ee0b7bea6399050da8271582c0381fb1930084d3ebd0867731809cbfd8bf5069"}
Nov 03 19:39:55.482219 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:55.481824    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:39:55.483466 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:55.483387    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerStarted","Data":"a5ae888670d147d2333357f46f183055d310272c3896cbf5baed60d208c08d6f"}
Nov 03 19:39:55.485960 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:55.485933    2572 generic.go:356] "Generic (PLEG): container finished" podID="5d178b87-95d6-4985-bf0f-f0448dd784d7" containerID="caf17b9b146ebca4fe9bdb860350e02b1a8041f319968de3e5adbc608df1e052" exitCode=0
Nov 03 19:39:55.486152 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:55.486131    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zftbb" event={"ID":"5d178b87-95d6-4985-bf0f-f0448dd784d7","Type":"ContainerDied","Data":"caf17b9b146ebca4fe9bdb860350e02b1a8041f319968de3e5adbc608df1e052"}
Nov 03 19:39:55.489378 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:55.489357    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d" event={"ID":"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1","Type":"ContainerStarted","Data":"436f2e3bb2dd117878aad180a08e2eed656551c4a4f7331ed21958caddcd7872"}
Nov 03 19:39:55.500828 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:55.500781    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zgwk6" podStartSLOduration=2.224674974 podStartE2EDuration="4.500766745s" totalImagesPullingTime="2.276091771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:55.500041206 +0000 UTC m=+50.801655695" watchObservedRunningTime="2026-11-03 19:39:55.500766745 +0000 UTC m=+50.802381235"
Nov 03 19:39:55.537950 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:55.537901    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d" podStartSLOduration=2.016068141 podStartE2EDuration="3.537884761s" totalImagesPullingTime="1.52181662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:55.536535983 +0000 UTC m=+50.838150475" watchObservedRunningTime="2026-11-03 19:39:55.537884761 +0000 UTC m=+50.839499247"
Nov 03 19:39:56.494701 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.494667    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62g8g" event={"ID":"2224bfc1-cc82-4e90-b723-e629d5f0f3f5","Type":"ContainerStarted","Data":"731fa49e0d73a9964363970dc262e43a00b20e289df6f16c8c037ecb766cb2f4"}
Nov 03 19:39:56.496261 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.496231    2572 generic.go:356] "Generic (PLEG): container finished" podID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerID="c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a" exitCode=0
Nov 03 19:39:56.496345 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.496307    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerDied","Data":"c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a"}
Nov 03 19:39:56.498408 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.498378    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btkpj" event={"ID":"8bedff4e-9e2b-4582-bafe-29360cfa83ae","Type":"ContainerStarted","Data":"6fc8bb045553ee506cc37859abf0eab2c2e5f4cd9d2c1023f46c66a8867cd7c1"}
Nov 03 19:39:56.500479 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.500460    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zftbb" event={"ID":"5d178b87-95d6-4985-bf0f-f0448dd784d7","Type":"ContainerStarted","Data":"15f8289bf94feddbd0991db1257dd4f5419345671d405e0b26fbc478e657927b"}
Nov 03 19:39:56.500565 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.500496    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zftbb" event={"ID":"5d178b87-95d6-4985-bf0f-f0448dd784d7","Type":"ContainerStarted","Data":"9ec08b95120ea8712d76e727b57ebf0a3ae9ead98582f14ac2ff62c37eb76d7a"}
Nov 03 19:39:56.502563 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.502543    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7" event={"ID":"edcd5516-336e-415d-b587-c887774d8a63","Type":"ContainerStarted","Data":"0db65e0e5f722611d4a74d1c2a49ae91f3cf1e2478162650865d4d95c988cc90"}
Nov 03 19:39:56.502648 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.502582    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7" event={"ID":"edcd5516-336e-415d-b587-c887774d8a63","Type":"ContainerStarted","Data":"f82a541a8277550f14d4844785a35c8b30dd9a9f729792bd816f9ceb389d9caa"}
Nov 03 19:39:56.502648 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.502600    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7" event={"ID":"edcd5516-336e-415d-b587-c887774d8a63","Type":"ContainerStarted","Data":"f796588d8bce7b22c629be8f578af00fe207ce10cfd8b9075446674f54bf5be2"}
Nov 03 19:39:56.510942 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.510905    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-62g8g" podStartSLOduration=1.572099898 podStartE2EDuration="5.510893886s" totalImagesPullingTime="3.938793988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:56.510126321 +0000 UTC m=+51.811740812" watchObservedRunningTime="2026-11-03 19:39:56.510893886 +0000 UTC m=+51.812508375"
Nov 03 19:39:56.527035 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.526964    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-btkpj" podStartSLOduration=2.470285804 podStartE2EDuration="5.526950839s" totalImagesPullingTime="3.056665035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:56.526105687 +0000 UTC m=+51.827720177" watchObservedRunningTime="2026-11-03 19:39:56.526950839 +0000 UTC m=+51.828565328"
Nov 03 19:39:56.544626 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.544579    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7" podStartSLOduration=2.604687378 podStartE2EDuration="4.544563412s" totalImagesPullingTime="1.939876034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:56.544344076 +0000 UTC m=+51.845958563" watchObservedRunningTime="2026-11-03 19:39:56.544563412 +0000 UTC m=+51.846177929"
Nov 03 19:39:56.564873 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:56.564824    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zftbb" podStartSLOduration=3.088838769 podStartE2EDuration="4.564807962s" totalImagesPullingTime="1.378313016s" totalInitContainerRuntime="97.656177ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:56.563935225 +0000 UTC m=+51.865549715" watchObservedRunningTime="2026-11-03 19:39:56.564807962 +0000 UTC m=+51.866422453"
Nov 03 19:39:57.315183 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.315143    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-cd7f87d47-wq4k7"]
Nov 03 19:39:57.318494 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.318470    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.322105 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.322082    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 19:39:57.322397 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.322376    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 19:39:57.332529 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.332507    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-cd7f87d47-wq4k7"]
Nov 03 19:39:57.337820 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.337797    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-client-ca-bundle\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.337922 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.337840    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-metrics-server-audit-profiles\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.337922 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.337874    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-tls\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.338020 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.337986    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c31a30a7-dcde-418e-8297-f5af07990cb0-audit-log\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.338087 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.338067    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m57w\" (UniqueName: \"kubernetes.io/projected/c31a30a7-dcde-418e-8297-f5af07990cb0-kube-api-access-9m57w\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.338201 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.338177    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.338285 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.338219    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-client-certs\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.360845 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.360773    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 19:39:57.361015 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.360980    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-67p1b6fhp1pgt\""
Nov 03 19:39:57.361109 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.361090    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 19:39:57.361214 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.361186    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2wh29\""
Nov 03 19:39:57.439182 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.439151    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c31a30a7-dcde-418e-8297-f5af07990cb0-audit-log\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.439360 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.439224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m57w\" (UniqueName: \"kubernetes.io/projected/c31a30a7-dcde-418e-8297-f5af07990cb0-kube-api-access-9m57w\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.439360 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.439261    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.439360 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.439286    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-client-certs\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.439360 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.439316    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-client-ca-bundle\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.439360 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.439347    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-metrics-server-audit-profiles\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.439626 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.439377    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-tls\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.439682 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.439618    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c31a30a7-dcde-418e-8297-f5af07990cb0-audit-log\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.440058 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.440028    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.440530 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.440493    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-metrics-server-audit-profiles\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.442309 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.442286    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-client-ca-bundle\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.442481 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.442425    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-tls\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.442794 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.442772    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-client-certs\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.447013 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.446988    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m57w\" (UniqueName: \"kubernetes.io/projected/c31a30a7-dcde-418e-8297-f5af07990cb0-kube-api-access-9m57w\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.632197 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.632103    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:39:57.914187 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:57.914161    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-cd7f87d47-wq4k7"]
Nov 03 19:39:57.916820 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:57.916781    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31a30a7_dcde_418e_8297_f5af07990cb0.slice/crio-c085aaf6c3736a7a8562e7defa661d2b376c6b508c5087b759117b2ab2ad6069 WatchSource:0}: Error finding container c085aaf6c3736a7a8562e7defa661d2b376c6b508c5087b759117b2ab2ad6069: Status 404 returned error can't find the container with id c085aaf6c3736a7a8562e7defa661d2b376c6b508c5087b759117b2ab2ad6069
Nov 03 19:39:58.512553 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:58.512458    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerStarted","Data":"ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae"}
Nov 03 19:39:58.512553 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:58.512539    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerStarted","Data":"895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf"}
Nov 03 19:39:58.512812 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:58.512568    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerStarted","Data":"d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9"}
Nov 03 19:39:58.512812 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:58.512594    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerStarted","Data":"cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c"}
Nov 03 19:39:58.512812 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:58.512625    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerStarted","Data":"e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5"}
Nov 03 19:39:58.513509 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:58.513488    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7" event={"ID":"c31a30a7-dcde-418e-8297-f5af07990cb0","Type":"ContainerStarted","Data":"c085aaf6c3736a7a8562e7defa661d2b376c6b508c5087b759117b2ab2ad6069"}
Nov 03 19:39:59.150009 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.149974    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:39:59.153668 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.153645    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.158874 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.158843    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 19:39:59.159004 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.158949    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 19:39:59.159052 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.159035    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 19:39:59.161425 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.159134    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 19:39:59.161425 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.159289    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d5pj80fngeohf\""
Nov 03 19:39:59.163621 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.163597    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 19:39:59.163753 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.163619    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 19:39:59.163753 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.163669    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 19:39:59.169717 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.169696    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 19:39:59.171346 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.171324    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 19:39:59.174379 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.174357    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:39:59.255801 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.255711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.255801 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.255793    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqrdl\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-kube-api-access-tqrdl\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256022 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.255858    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256022 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.255901    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256022 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.255931    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256022 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.255957    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256208 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256026    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256208 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256097    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256208 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256129    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256208 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256193    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256395 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256395 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256280    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256395 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256297    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256395 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256337    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256395 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256355    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-config\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256395 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256370    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256698 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256479    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256698 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256505    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256698 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256521    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.256698 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.256539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.259024 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.259002    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 19:39:59.259024 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.259003    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 19:39:59.259218 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.259077    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 19:39:59.259218 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.259119    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x7mbg\""
Nov 03 19:39:59.259218 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.259189    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 19:39:59.259359 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.259237    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 19:39:59.357201 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357169    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrdl\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-kube-api-access-tqrdl\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357201 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357207    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357408 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357225    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357408 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357408 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357283    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357408 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357316    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357408 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357342    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357408 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357362    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357408 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357786 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357420    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357786 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357497    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357786 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357522    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357786 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357544    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357786 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357610    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-config\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357786 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357643    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357786 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357678    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357786 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357710    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357786 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357732    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.357786 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357761    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.358274 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357799    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.358274 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.357834    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.358380 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.358270    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.358380 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.358291    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.358894 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.358869    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.361062 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.360424    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.361062 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.360754    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.361062 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.360949    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.361464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.361423    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.361464 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.360754    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.362492 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.362184    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.362492 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.362266    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.362492 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.362399    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.362738 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.362715    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-config\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.362912 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.362880    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.363766 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.363673    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.363850 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.363782    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.364407 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.364378    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.364546 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.364528    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.364843 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.364822    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.365953 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.365936    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrdl\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-kube-api-access-tqrdl\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.482665 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.482637    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:59.521799 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.521696    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7" event={"ID":"c31a30a7-dcde-418e-8297-f5af07990cb0","Type":"ContainerStarted","Data":"ecf44097f4c00fd26a54c4a6ce01ecf712064ff267d35fe28d2aa9f5ed6840e9"}
Nov 03 19:39:59.526210 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.526181    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerStarted","Data":"423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a"}
Nov 03 19:39:59.571453 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.571367    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7" podStartSLOduration=1.073612986 podStartE2EDuration="2.57134729s" totalImagesPullingTime="1.497734304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:59.544580117 +0000 UTC m=+54.846194608" watchObservedRunningTime="2026-11-03 19:39:59.57134729 +0000 UTC m=+54.872961781"
Nov 03 19:39:59.572048 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.572013    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.190200861 podStartE2EDuration="6.572001111s" totalImagesPullingTime="3.368945549s" totalInitContainerRuntime="12.854701ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:59.570679763 +0000 UTC m=+54.872294253" watchObservedRunningTime="2026-11-03 19:39:59.572001111 +0000 UTC m=+54.873615602"
Nov 03 19:39:59.770820 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:39:59.770777    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:39:59.772840 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:39:59.772782    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7d54f2_9efe_467f_82a4_693e30e17b58.slice/crio-cb232ad37c2c4153751d97b8e4c603375bdb7d55824e2f98d0005baec81a3471 WatchSource:0}: Error finding container cb232ad37c2c4153751d97b8e4c603375bdb7d55824e2f98d0005baec81a3471: Status 404 returned error can't find the container with id cb232ad37c2c4153751d97b8e4c603375bdb7d55824e2f98d0005baec81a3471
Nov 03 19:40:00.535384 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:00.535348    2572 generic.go:356] "Generic (PLEG): container finished" podID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerID="e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f" exitCode=0
Nov 03 19:40:00.535865 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:00.535447    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerDied","Data":"e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f"}
Nov 03 19:40:00.535865 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:00.535555    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerStarted","Data":"cb232ad37c2c4153751d97b8e4c603375bdb7d55824e2f98d0005baec81a3471"}
Nov 03 19:40:05.505456 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:05.505413    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:40:05.557080 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:05.557045    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerStarted","Data":"5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2"}
Nov 03 19:40:05.557240 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:05.557129    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerStarted","Data":"b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118"}
Nov 03 19:40:06.237397 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:06.237371    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:08.570675 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:08.570642    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerStarted","Data":"15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686"}
Nov 03 19:40:08.571081 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:08.570703    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerStarted","Data":"37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1"}
Nov 03 19:40:08.571081 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:08.570729    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerStarted","Data":"c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6"}
Nov 03 19:40:08.571081 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:08.570754    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerStarted","Data":"6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a"}
Nov 03 19:40:08.599118 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:08.599065    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.184927928 podStartE2EDuration="9.599037248s" totalImagesPullingTime="7.398689079s" totalInitContainerRuntime="15.420241ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:08.597272559 +0000 UTC m=+63.898887049" watchObservedRunningTime="2026-11-03 19:40:08.599037248 +0000 UTC m=+63.900651740"
Nov 03 19:40:09.483487 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:09.483455    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:10.962787 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:10.962750    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:40:10.962787 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:10.962789    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:40:10.965627 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:10.965609    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:40:10.966787 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:10.966771    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:40:10.976443 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:10.976409    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:40:10.976568 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:10.976411    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:40:11.001483 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.001463    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5rk8\""
Nov 03 19:40:11.008494 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.008474    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:40:11.164450 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.164408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:40:11.167176 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.167159    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:40:11.177801 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.177782    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:40:11.188579 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.188561    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:40:11.276662 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.276585    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mr2m2"]
Nov 03 19:40:11.278622 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.278600    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-drbgz\""
Nov 03 19:40:11.279618 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:40:11.279596    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3bbe34_cf5f_4d4f_88b9_600903ab0961.slice/crio-9b69f39eef492a47cb2954c2dc9b5d18157ed2e092eccb392f9db0ab015b0f96 WatchSource:0}: Error finding container 9b69f39eef492a47cb2954c2dc9b5d18157ed2e092eccb392f9db0ab015b0f96: Status 404 returned error can't find the container with id 9b69f39eef492a47cb2954c2dc9b5d18157ed2e092eccb392f9db0ab015b0f96
Nov 03 19:40:11.286297 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.286279    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:40:11.315240 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.315217    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zcfm6\""
Nov 03 19:40:11.323317 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.323294    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:40:11.555068 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.555046    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9sbm7"]
Nov 03 19:40:11.556878 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:40:11.556852    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2aab3d5_357d_470a_a27d_bab9d570a73d.slice/crio-ce8887671e75021e0a061d1ded1eece177ef4a90b77b4a2d496fa15b1e9cb0c0 WatchSource:0}: Error finding container ce8887671e75021e0a061d1ded1eece177ef4a90b77b4a2d496fa15b1e9cb0c0: Status 404 returned error can't find the container with id ce8887671e75021e0a061d1ded1eece177ef4a90b77b4a2d496fa15b1e9cb0c0
Nov 03 19:40:11.581109 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.581082    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mr2m2" event={"ID":"eb3bbe34-cf5f-4d4f-88b9-600903ab0961","Type":"ContainerStarted","Data":"9b69f39eef492a47cb2954c2dc9b5d18157ed2e092eccb392f9db0ab015b0f96"}
Nov 03 19:40:11.582164 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.582142    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9sbm7" event={"ID":"e2aab3d5-357d-470a-a27d-bab9d570a73d","Type":"ContainerStarted","Data":"ce8887671e75021e0a061d1ded1eece177ef4a90b77b4a2d496fa15b1e9cb0c0"}
Nov 03 19:40:11.598760 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:11.598704    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9bwkj"]
Nov 03 19:40:11.601210 ip-10-0-128-24 kubenswrapper[2572]: W1103 19:40:11.601181    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70bada2_ec8a_40db_80d9_6455a9a675a3.slice/crio-b3313a9fa2af18dd113f4cd4b3572b019043aba71daa7c3420db469c62537dee WatchSource:0}: Error finding container b3313a9fa2af18dd113f4cd4b3572b019043aba71daa7c3420db469c62537dee: Status 404 returned error can't find the container with id b3313a9fa2af18dd113f4cd4b3572b019043aba71daa7c3420db469c62537dee
Nov 03 19:40:12.589496 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:12.589385    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mr2m2" event={"ID":"eb3bbe34-cf5f-4d4f-88b9-600903ab0961","Type":"ContainerStarted","Data":"dce7dd125e694e529ab2078c7385b757dd4a36b0c556e7615eeb937c7f4ca9eb"}
Nov 03 19:40:12.591155 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:12.591129    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9bwkj" event={"ID":"d70bada2-ec8a-40db-80d9-6455a9a675a3","Type":"ContainerStarted","Data":"b3313a9fa2af18dd113f4cd4b3572b019043aba71daa7c3420db469c62537dee"}
Nov 03 19:40:13.597034 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:13.596993    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mr2m2" event={"ID":"eb3bbe34-cf5f-4d4f-88b9-600903ab0961","Type":"ContainerStarted","Data":"5c756abe6e8e584a879dfa7f5faa8f51dc1458db7238983c28e24ddaf8903da6"}
Nov 03 19:40:13.615482 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:13.615403    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mr2m2" podStartSLOduration=66.509784612 podStartE2EDuration="1m7.61538353s" totalImagesPullingTime="1.105598918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:13.614524402 +0000 UTC m=+68.916138893" watchObservedRunningTime="2026-11-03 19:40:13.61538353 +0000 UTC m=+68.916998023"
Nov 03 19:40:15.607034 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:15.606992    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9bwkj" event={"ID":"d70bada2-ec8a-40db-80d9-6455a9a675a3","Type":"ContainerStarted","Data":"21dede008f526e3568f8f09948e544f30b8ea871efa07104c3183bb0f02ae61c"}
Nov 03 19:40:15.607525 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:15.607139    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:40:15.623532 ip-10-0-128-24 kubenswrapper[2572]: I1103 19:40:15.623482    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9bwkj" podStartSLOduration=66.431935831 podStartE2EDuration="1m9.623468643s" totalImagesPullingTime="3.191532812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:15.62259038 +0000 UTC m=+70.924204875" watchObservedRunningTime="2026-11-03 19:40:15.623468643 +0000 UTC m=+70.925083133"
Nov 03 19:40:17.413566 ip-10-0-128-24 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 19:40:17.431821 ip-10-0-128-24 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 19:40:17.431993 ip-10-0-128-24 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 19:40:17.432171 ip-10-0-128-24 systemd[1]: kubelet.service: Consumed 4.273s CPU time.
Nov 03 19:40:17.451238 ip-10-0-128-24 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:40:17.516111 ip-10-0-128-24 kubenswrapper[5600]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:40:17.516111 ip-10-0-128-24 kubenswrapper[5600]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:40:17.516111 ip-10-0-128-24 kubenswrapper[5600]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:40:17.516111 ip-10-0-128-24 kubenswrapper[5600]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:40:17.518538 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518446    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:17.518538 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518532    5600 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:17.518538 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518538    5600 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:17.518538 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518541    5600 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:17.518538 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518545    5600 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518550    5600 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518555    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518559    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518562    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518566    5600 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518570    5600 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518574    5600 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518577    5600 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518581    5600 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518585    5600 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518589    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518593    5600 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518597    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518601    5600 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518604    5600 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518608    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518612    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518615    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:17.518849 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518620    5600 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518623    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518627    5600 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518631    5600 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518645    5600 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518650    5600 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518655    5600 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518659    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518663    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518666    5600 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518671    5600 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518675    5600 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518679    5600 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518683    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518687    5600 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518693    5600 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518697    5600 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518702    5600 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518707    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518711    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:17.519691 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518716    5600 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518720    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518725    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518730    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518734    5600 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518738    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518743    5600 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518747    5600 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518751    5600 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518755    5600 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518759    5600 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518764    5600 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518770    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518774    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518779    5600 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518785    5600 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518789    5600 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518793    5600 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518798    5600 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518802    5600 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:17.520563 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518807    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518811    5600 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518815    5600 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518819    5600 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518825    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518829    5600 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518833    5600 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518840    5600 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518847    5600 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518851    5600 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518856    5600 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518862    5600 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518866    5600 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518871    5600 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518877    5600 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518881    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518887    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518891    5600 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518895    5600 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518900    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:17.521391 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518904    5600 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518908    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518912    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518917    5600 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518921    5600 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518925    5600 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518929    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518937    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518942    5600 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518947    5600 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518951    5600 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518955    5600 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518959    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518964    5600 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518968    5600 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518973    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518977    5600 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518981    5600 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518990    5600 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:17.522472 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.518997    5600 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.519001    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.519006    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.519010    5600 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520243    5600 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520253    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520257    5600 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520262    5600 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520266    5600 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520270    5600 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520275    5600 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520279    5600 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520283    5600 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520288    5600 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520293    5600 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520297    5600 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520301    5600 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520305    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520309    5600 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:17.523336 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520314    5600 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520319    5600 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520323    5600 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520328    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520332    5600 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520337    5600 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520341    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520345    5600 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520349    5600 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520353    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520358    5600 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520362    5600 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520367    5600 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520371    5600 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520376    5600 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520380    5600 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520384    5600 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520388    5600 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520392    5600 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520397    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:17.524211 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520401    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520405    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520409    5600 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520414    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520419    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520423    5600 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520427    5600 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520451    5600 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520455    5600 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520459    5600 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520463    5600 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520467    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520471    5600 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520475    5600 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520479    5600 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520483    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520488    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520493    5600 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520497    5600 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520501    5600 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:17.524970 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520506    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520510    5600 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520514    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520518    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520522    5600 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520526    5600 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520531    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520535    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520539    5600 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520543    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520548    5600 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520552    5600 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520556    5600 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520560    5600 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520564    5600 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520568    5600 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520572    5600 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520576    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520582    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:17.525481 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520587    5600 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520593    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520598    5600 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520603    5600 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520608    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520613    5600 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520618    5600 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520622    5600 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520626    5600 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520630    5600 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520635    5600 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520640    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520645    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520649    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520654    5600 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520658    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520662    5600 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520666    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520670    5600 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520674    5600 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:17.525933 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520679    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520683    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520687    5600 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520692    5600 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520696    5600 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520700    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520704    5600 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520709    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520715    5600 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520721    5600 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520727    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.520732    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521526    5600 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521536    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521540    5600 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521544    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521548    5600 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521552    5600 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521556    5600 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:17.526493 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521560    5600 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521564    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521569    5600 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521574    5600 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521578    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521582    5600 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521587    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521591    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521595    5600 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521602    5600 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521606    5600 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521610    5600 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521614    5600 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521618    5600 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521622    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521625    5600 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521629    5600 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521633    5600 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521636    5600 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521640    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:17.527118 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521645    5600 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521649    5600 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521653    5600 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521657    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521661    5600 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521666    5600 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521670    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521674    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521678    5600 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521682    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521687    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521691    5600 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521695    5600 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521700    5600 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521705    5600 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521709    5600 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521713    5600 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521718    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521722    5600 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:17.527930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521726    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521730    5600 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521736    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521740    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521746    5600 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521752    5600 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521756    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521760    5600 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521764    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521769    5600 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521772    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521777    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521782    5600 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521786    5600 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521790    5600 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521794    5600 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521798    5600 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521802    5600 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521806    5600 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:17.528425 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521811    5600 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521817    5600 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521823    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521827    5600 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521832    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521836    5600 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521840    5600 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521844    5600 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521848    5600 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521852    5600 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521856    5600 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521861    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521865    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521870    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521876    5600 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521882    5600 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521886    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521890    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521894    5600 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521899    5600 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:17.529080 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521903    5600 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521907    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521911    5600 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521915    5600 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521919    5600 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521925    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521929    5600 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521933    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521937    5600 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521941    5600 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521945    5600 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521949    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521953    5600 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521957    5600 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521961    5600 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521965    5600 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521969    5600 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521973    5600 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521978    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521981    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:17.529580 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.521985    5600 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522127    5600 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522144    5600 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522154    5600 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522162    5600 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522169    5600 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522176    5600 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522184    5600 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522191    5600 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522196    5600 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522201    5600 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522207    5600 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522212    5600 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522217    5600 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522222    5600 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522226    5600 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522231    5600 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522235    5600 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522239    5600 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522247    5600 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522251    5600 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522256    5600 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522261    5600 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522267    5600 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:40:17.530060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522273    5600 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522277    5600 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522282    5600 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522287    5600 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522292    5600 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522298    5600 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522302    5600 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522307    5600 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522312    5600 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522318    5600 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522323    5600 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522328    5600 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522333    5600 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522338    5600 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522345    5600 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522353    5600 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522358    5600 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522363    5600 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522367    5600 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522373    5600 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522379    5600 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522384    5600 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522389    5600 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522394    5600 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522399    5600 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:40:17.530662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522403    5600 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522408    5600 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522412    5600 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522417    5600 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522421    5600 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522426    5600 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522452    5600 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522457    5600 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522462    5600 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522467    5600 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522471    5600 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522476    5600 flags.go:64] FLAG: --help="false"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522480    5600 flags.go:64] FLAG: --hostname-override="ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522485    5600 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522489    5600 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522494    5600 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522499    5600 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522505    5600 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522509    5600 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522514    5600 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522519    5600 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522526    5600 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522532    5600 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:40:17.531260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522537    5600 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522542    5600 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522547    5600 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522551    5600 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522556    5600 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522561    5600 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522565    5600 flags.go:64] FLAG: --lock-file=""
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522570    5600 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522574    5600 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522579    5600 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522588    5600 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522593    5600 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522598    5600 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522602    5600 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522607    5600 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522612    5600 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522617    5600 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522621    5600 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522628    5600 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522633    5600 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522639    5600 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522644    5600 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522649    5600 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522654    5600 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522658    5600 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:40:17.531869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522663    5600 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522667    5600 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522672    5600 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522683    5600 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522688    5600 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522695    5600 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522702    5600 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522707    5600 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522711    5600 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522716    5600 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522720    5600 flags.go:64] FLAG: --port="10250"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522725    5600 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522730    5600 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0fe8f0d9aec6eeb9a"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522735    5600 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522740    5600 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522745    5600 flags.go:64] FLAG: --register-node="true"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522749    5600 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522755    5600 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522760    5600 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522765    5600 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522769    5600 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522776    5600 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522780    5600 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522785    5600 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522790    5600 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522794    5600 flags.go:64] FLAG: --runonce="false"
Nov 03 19:40:17.532630 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522799    5600 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522804    5600 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522809    5600 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522813    5600 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522818    5600 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522822    5600 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522827    5600 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522831    5600 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522837    5600 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522841    5600 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522846    5600 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522853    5600 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522859    5600 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522865    5600 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522869    5600 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522877    5600 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522883    5600 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522887    5600 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522894    5600 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522898    5600 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522902    5600 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522907    5600 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522911    5600 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522916    5600 flags.go:64] FLAG: --v="2"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522933    5600 flags.go:64] FLAG: --version="false"
Nov 03 19:40:17.533238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522940    5600 flags.go:64] FLAG: --vmodule=""
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522946    5600 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.522951    5600 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523158    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523165    5600 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523170    5600 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523178    5600 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523184    5600 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523188    5600 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523193    5600 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523197    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523201    5600 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523205    5600 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523209    5600 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523214    5600 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523218    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523222    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523226    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523232    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523240    5600 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:17.533870 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523244    5600 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523249    5600 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523253    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523258    5600 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523262    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523266    5600 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523270    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523275    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523279    5600 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523283    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523288    5600 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523292    5600 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523296    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523301    5600 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523306    5600 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523310    5600 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523314    5600 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523318    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523324    5600 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:17.534491 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523331    5600 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523336    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523340    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523345    5600 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523350    5600 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523353    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523357    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523361    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523366    5600 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523370    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523374    5600 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523380    5600 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523385    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523389    5600 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523393    5600 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523399    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523403    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523407    5600 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523412    5600 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523416    5600 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:17.535063 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523420    5600 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523425    5600 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523450    5600 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523454    5600 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523459    5600 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523463    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523467    5600 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523471    5600 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523475    5600 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523480    5600 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523484    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523488    5600 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523492    5600 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523497    5600 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523501    5600 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523505    5600 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523509    5600 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523513    5600 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523517    5600 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:17.535762 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523521    5600 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523524    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523529    5600 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523533    5600 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523540    5600 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523544    5600 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523548    5600 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523552    5600 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523556    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523561    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523565    5600 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523570    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523574    5600 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523578    5600 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523582    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523586    5600 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523590    5600 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523594    5600 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523599    5600 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523603    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:17.536272 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523606    5600 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523611    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523615    5600 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523619    5600 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523623    5600 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523628    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523632    5600 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523636    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523641    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523645    5600 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523649    5600 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.523693    5600 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523749    5600 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523756    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523761    5600 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:17.536840 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523765    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523771    5600 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523775    5600 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523780    5600 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523784    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523788    5600 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523793    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523798    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523802    5600 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523806    5600 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523810    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523814    5600 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523818    5600 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523824    5600 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523830    5600 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523835    5600 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523840    5600 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523844    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523848    5600 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:17.537324 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523852    5600 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523856    5600 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523860    5600 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523865    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523869    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523873    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523877    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523881    5600 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523885    5600 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523890    5600 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523894    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523898    5600 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523902    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523906    5600 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523913    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523918    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523923    5600 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523927    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523931    5600 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523935    5600 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:17.537881 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523940    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523945    5600 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523949    5600 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523953    5600 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523957    5600 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523967    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523974    5600 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523981    5600 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523986    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523991    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.523995    5600 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524000    5600 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524005    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524009    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524013    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524018    5600 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524022    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524026    5600 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524030    5600 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:17.538423 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524035    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524039    5600 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524043    5600 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524047    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524051    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524055    5600 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524060    5600 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524067    5600 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524072    5600 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524076    5600 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524080    5600 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524084    5600 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524089    5600 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524093    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524097    5600 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524103    5600 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524107    5600 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524111    5600 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524115    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524119    5600 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:17.538907 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524123    5600 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524127    5600 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524132    5600 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524136    5600 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524140    5600 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524145    5600 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524149    5600 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524153    5600 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524157    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524162    5600 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524166    5600 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524170    5600 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524174    5600 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524178    5600 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524182    5600 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524187    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524191    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524196    5600 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524200    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524207    5600 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:17.539382 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524212    5600 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524215    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524219    5600 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524223    5600 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.524227    5600 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.529493    5600 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.529510    5600 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.529528    5600 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.529534    5600 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529590    5600 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529596    5600 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529600    5600 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529603    5600 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529607    5600 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529609    5600 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529612    5600 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529615    5600 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529617    5600 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529620    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529622    5600 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:17.539925 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529625    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529627    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529630    5600 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529632    5600 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529636    5600 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529639    5600 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529641    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529644    5600 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529647    5600 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529649    5600 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529652    5600 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529654    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529656    5600 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529659    5600 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529661    5600 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529664    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529667    5600 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529669    5600 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529672    5600 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529674    5600 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:17.540405 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529677    5600 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529679    5600 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529682    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529684    5600 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529689    5600 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529692    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529695    5600 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529698    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529700    5600 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529703    5600 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529705    5600 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529708    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529711    5600 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529713    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529716    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529718    5600 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529720    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529723    5600 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529725    5600 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:17.540930 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529728    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529730    5600 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529733    5600 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529735    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529738    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529741    5600 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529743    5600 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529746    5600 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529748    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529751    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529753    5600 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529756    5600 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529758    5600 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529761    5600 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529764    5600 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529767    5600 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529771    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529774    5600 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529776    5600 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529779    5600 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:17.541389 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529781    5600 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529784    5600 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529786    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529788    5600 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529791    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529794    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529796    5600 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529799    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529801    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529804    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529807    5600 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529809    5600 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529813    5600 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529816    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529818    5600 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529821    5600 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529823    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529826    5600 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529829    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:17.541882 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529831    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529833    5600 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529836    5600 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529838    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529841    5600 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529844    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529846    5600 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529849    5600 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529851    5600 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529854    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529856    5600 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529859    5600 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529861    5600 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529864    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529866    5600 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529869    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:17.542366 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.529871    5600 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.529899    5600 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530046    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530051    5600 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530054    5600 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530057    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530059    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530062    5600 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530065    5600 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530068    5600 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530071    5600 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530073    5600 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530076    5600 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530079    5600 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530091    5600 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530094    5600 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:17.542774 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530097    5600 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530100    5600 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530102    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530105    5600 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530107    5600 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530110    5600 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530112    5600 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530115    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530118    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530121    5600 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530123    5600 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530125    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530128    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530131    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530133    5600 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530136    5600 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530138    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530141    5600 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530143    5600 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:17.543156 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530146    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530149    5600 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530153    5600 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530155    5600 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530157    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530163    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530165    5600 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530168    5600 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530170    5600 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530173    5600 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530175    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530178    5600 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530180    5600 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530183    5600 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530186    5600 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530190    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530192    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530195    5600 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530197    5600 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:17.543620 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530200    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530202    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530205    5600 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530207    5600 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530210    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530212    5600 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530215    5600 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530217    5600 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530220    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530222    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530224    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530227    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530229    5600 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530232    5600 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530235    5600 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530237    5600 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530239    5600 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530242    5600 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530245    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:17.544144 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530248    5600 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530250    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530253    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530255    5600 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530257    5600 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530260    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530262    5600 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530265    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530267    5600 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530269    5600 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530272    5600 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530274    5600 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530277    5600 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530279    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530282    5600 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530284    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530287    5600 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530289    5600 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530292    5600 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530294    5600 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:17.544803 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530296    5600 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530299    5600 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530301    5600 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530304    5600 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530306    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530308    5600 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530311    5600 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530313    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530315    5600 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530318    5600 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530320    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530323    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530325    5600 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530329    5600 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530331    5600 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.530354    5600 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:40:17.545370 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530386    5600 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530390    5600 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530393    5600 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530396    5600 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530398    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530401    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530404    5600 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530406    5600 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530411    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530414    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530416    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530419    5600 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530421    5600 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530423    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530426    5600 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530442    5600 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530445    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530449    5600 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530452    5600 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:17.545884 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530455    5600 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530458    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530460    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530463    5600 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530465    5600 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530468    5600 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530470    5600 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530474    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530476    5600 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530478    5600 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530481    5600 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530486    5600 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530489    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530493    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530495    5600 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530498    5600 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530501    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530503    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530506    5600 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:17.546610 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530509    5600 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530511    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530515    5600 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530518    5600 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530520    5600 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530523    5600 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530525    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530528    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530530    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530533    5600 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530535    5600 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530539    5600 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530541    5600 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530544    5600 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530546    5600 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530549    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530551    5600 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530554    5600 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530556    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:17.547469 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530559    5600 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530561    5600 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530564    5600 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530566    5600 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530569    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530571    5600 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530574    5600 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530577    5600 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530579    5600 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530581    5600 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530584    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530586    5600 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530588    5600 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530591    5600 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530594    5600 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530597    5600 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530600    5600 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530602    5600 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530605    5600 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530608    5600 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:17.548007 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530610    5600 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530613    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530615    5600 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530617    5600 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530620    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530622    5600 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530625    5600 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530627    5600 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530630    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530632    5600 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530635    5600 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530637    5600 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530640    5600 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530642    5600 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530645    5600 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530647    5600 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530649    5600 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530652    5600 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530654    5600 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530658    5600 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:17.548586 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530660    5600 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530663    5600 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530665    5600 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530668    5600 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530671    5600 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530673    5600 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530676    5600 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530680    5600 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:17.530682    5600 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.530740    5600 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.531709    5600 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.531769    5600 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.531944    5600 server.go:1014] "Starting client certificate rotation"
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.532082    5600 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.532126    5600 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:34:06 +0000 UTC" deadline="2028-06-07 08:54:38.542564801 +0000 UTC"
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.532157    5600 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13957h14m21.010409966s"
Nov 03 19:40:17.549101 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.532404    5600 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:40:17.549545 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.533473    5600 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:40:17.549545 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.534840    5600 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:40:17.549545 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.536654    5600 log.go:25] "Validated CRI v1 image API"
Nov 03 19:40:17.549545 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.537484    5600 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:40:17.549545 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.539628    5600 fs.go:135] Filesystem UUIDs: map[63d116ec-017e-4fb1-b6e1-e15a9c9a4f56:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c769c5d2-a8eb-46cd-823d-42bdc89e9959:/dev/nvme0n1p4]
Nov 03 19:40:17.549699 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.539647    5600 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b6f018189450cd956b8b25de13d8b2f5c6c0f236fd7835b69315c4840131a8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b6f018189450cd956b8b25de13d8b2f5c6c0f236fd7835b69315c4840131a8a/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21752a7a0138e2305b6882d40ed02003ee0660c837bbfa7a2c48283d107a22be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21752a7a0138e2305b6882d40ed02003ee0660c837bbfa7a2c48283d107a22be/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/246af0f3d3dd0a114e9cdca6989f71416ae7e97c98ef154a92ddb038288644b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/246af0f3d3dd0a114e9cdca6989f71416ae7e97c98ef154a92ddb038288644b6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27b11c5c6597d0250624a1141508bb56fd612ede51505a8053d7445197e92a6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27b11c5c6597d0250624a1141508bb56fd612ede51505a8053d7445197e92a6b/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39b2397af6c1565c258e80f46815d70fe4d820256d5b76c46112a390b842ade8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39b2397af6c1565c258e80f46815d70fe4d820256d5b76c46112a390b842ade8/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48f72ab27ab404e9d978dfaf7e3d68a71d5db54d62bddf7f4cf8743da250db58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48f72ab27ab404e9d978dfaf7e3d68a71d5db54d62bddf7f4cf8743da250db58/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55ed9d78bcef62604e945b7540a963036fb859a46533d9904b9c429d5d0b2048/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55ed9d78bcef62604e945b7540a963036fb859a46533d9904b9c429d5d0b2048/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6678bac546496281365a97a349a3cd41a1bc6a2055a4753a0ebb8f2dfe921f34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6678bac546496281365a97a349a3cd41a1bc6a2055a4753a0ebb8f2dfe921f34/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ae954350a2552cfef72be13db2b78f3546858a6c9221b01e1a27213b3c791bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ae954350a2552cfef72be13db2b78f3546858a6c9221b01e1a27213b3c791bd/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b69f39eef492a47cb2954c2dc9b5d18157ed2e092eccb392f9db0ab015b0f96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b69f39eef492a47cb2954c2dc9b5d18157ed2e092eccb392f9db0ab015b0f96/userdata/shm major:0 minor:421 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5ae888670d147d2333357f46f183055d310272c3896cbf5baed60d208c08d6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5ae888670d147d2333357f46f183055d310272c3896cbf5baed60d208c08d6f/userdata/shm major:0 minor:283 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa90567a31bf21c64a5651c62d660a07942d209f34ee252ff8a214d4e389da57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa90567a31bf21c64a5651c62d660a07942d209f34ee252ff8a214d4e389da57/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad6d6c671a0cc6d8d687fcf5cf5f30dd3f1661857d4d9bd1488991d60f18c718/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad6d6c671a0cc6d8d687fcf5cf5f30dd3f1661857d4d9bd1488991d60f18c718/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3313a9fa2af18dd113f4cd4b3572b019043aba71daa7c3420db469c62537dee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3313a9fa2af18dd113f4cd4b3572b019043aba71daa7c3420db469c62537dee/userdata/shm major:0 minor:428 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b587f04ed3984d763957dd90913c8bc07819228a05c9e8749b16df29c9299ff3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b587f04ed3984d763957dd90913c8bc07819228a05c9e8749b16df29c9299ff3/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c085aaf6c3736a7a8562e7defa661d2b376c6b508c5087b759117b2ab2ad6069/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c085aaf6c3736a7a8562e7defa661d2b376c6b508c5087b759117b2ab2ad6069/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb232ad37c2c4153751d97b8e4c603375bdb7d55824e2f98d0005baec81a3471/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb232ad37c2c4153751d97b8e4c603375bdb7d55824e2f98d0005baec81a3471/userdata/shm major:0 minor:380 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc72b4ca04bc904a8d9e4c670bc021fe46d1623d6e1bdf0fb2b65e77b6736fdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc72b4ca04bc904a8d9e4c670bc021fe46d1623d6e1bdf0fb2b65e77b6736fdb/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce8887671e75021e0a061d1ded1eece177ef4a90b77b4a2d496fa15b1e9cb0c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce8887671e75021e0a061d1ded1eece177ef4a90b77b4a2d496fa15b1e9cb0c0/userdata/shm major:0 minor:426 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf1d6b248740da36f5e293f6636bbf18e1c3da50d6144b5b437a9c702063cc51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf1d6b248740da36f5e293f6636bbf18e1c3da50d6144b5b437a9c702063cc51/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd674b248c6165f5d22a07c5a067c2db45322c3d1a86dcf72083d284a8d16775/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd674b248c6165f5d22a07c5a067c2db45322c3d1a86dcf72083d284a8d16775/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7f7e72c047d1f63fa136e8bdf58d06b70d33d94913245861076f11e7eeadedb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7f7e72c047d1f63fa136e8bdf58d06b70d33d94913245861076f11e7eeadedb/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0034d15b-8b67-4e49-8abd-b834b164af4f/volumes/kubernetes.io~projected/kube-api-access-slxrz:{mountpoint:/var/lib/kubelet/pods/0034d15b-8b67-4e49-8abd-b834b164af4f/volumes/kubernetes.io~projected/kube-api-access-slxrz major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16525d8b-c267-41c1-8048-cec8f8dc381f/volumes/kubernetes.io~projected/kube-api-access-88nc2:{mountpoint:/var/lib/kubelet/pods/16525d8b-c267-41c1-8048-cec8f8dc381f/volumes/kubernetes.io~projected/kube-api-access-88nc2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/web-config/alertmanager/9 major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/web-config/config-reloader/6 major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~empty-dir/config-out major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~projected/kube-api-access-s5qzh:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~projected/kube-api-access-s5qzh major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~projected/tls-assets major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/config-volume major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/web-config major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2224bfc1-cc82-4e90-b723-e629d5f0f3f5/volumes/kubernetes.io~projected/kube-api-access-rqvvg:{mountpoint:/var/lib/kubelet/pods/2224bfc1-cc82-4e90-b723-e629d5f0f3f5/volumes/kubernetes.io~projected/kube-api-access-rqvvg major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2224bfc1-cc82-4e90-b723-e629d5f0f3f5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2224bfc1-cc82-4e90-b723-e629d5f0f3f5/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fde71af-248d-498d-b358-4db5a52a8cd0/volumes/kubernetes.io~projected/kube-api-access-qlxts:{mountpoint:/var/lib/kubelet/pods/2fde71af-248d-498d-b358-4db5a52a8cd0/volumes/kubernetes.io~projected/kube-api-access-qlxts major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25/volumes/kubernetes.io~projected/kube-api-access-kpp59:{mountpoint:/var/lib/kubelet/pods/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25/volumes/kubernetes.io~projected/kube-api-access-kpp59 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/576764d6-ba4a-416e-aac2-13e08c76680a/volumes/kubernetes.io~projected/kube-api-access-2h8jg:{mountpoint:/var/lib/kubelet/pods/576764d6-ba4a-416e-aac2-13e08c76680a/volumes/kubernetes.io~projected/kube-api-access-2h8jg major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d178b87-95d6-4985-bf0f-f0448dd784d7/volumes/kubernetes.io~projected/kube-api-access-k6rp8:{mountpoint:/var/lib/kubelet/pods/5d178b87-95d6-4985-bf0f-f0448dd784d7/volumes/kubernetes.io~projected/kube-api-access-k6rp8 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d178b87-95d6-4985-bf0f-f0448dd784d7/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5d178b87-95d6-4985-bf0f-f0448dd784d7/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d178b87-95d6-4985-bf0f-f0448dd784d7/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/5d178b87-95d6-4985-bf0f-f0448dd784d7/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81c1c0da-1ec1-4a75-9017-430bcedcc19b/volumes/kubernetes.io~projected/kube-api-access-4znh8:{mountpoint:/var/lib/kubelet/pods/81c1c0da-1ec1-4a75-9017-430bcedcc19b/volumes/kubernetes.io~projected/kube-api-access-4znh8 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8668d2b7-8bf0-4786-b96f-0ba9ab45472c/volumes/kubernetes.io~projected/kube-api-access-bphw8:{mountpoint:/var/lib/kubelet/pods/8668d2b7-8bf0-4786-b96f-0ba9ab45472c/volumes/kubernetes.io~projected/kube-api-access-bphw8 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8668d2b7-8bf0-4786-b96f-0ba9ab45472c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/8668d2b7-8bf0-4786-b96f-0ba9ab45472c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bedff4e-9e2b-4582-bafe-29360cfa83ae/volumes/kubernetes.io~projected/kube-api-access-v8jnr:{mountpoint:/var/lib/kubelet/pods/8bedff4e-9e2b-4582-bafe-29360cfa83ae/volumes/kubernetes.io~projected/kube-api-access-v8jnr major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bedff4e-9e2b-4582-bafe-29360cfa83ae/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/8bedff4e-9e2b-4582-bafe-29360cfa83ae/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volume-subpaths/web-config/config-reloader/2 major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volume-subpaths/web-config/prometheus/15 major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~empty-dir/config-out major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~projected/kube-api-access-tqrdl:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~projected/kube-api-access-tqrdl major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~projected/tls-assets major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/config major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/web-config major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2ca8b68-6873-4723-ad8f-92876443f8fa/volumes/kubernetes.io~projected/kube-api-access-qpmh6:{mountpoint:/var/lib/kubelet/pods/a2ca8b68-6873-4723-ad8f-92876443f8fa/volumes/kubernetes.io~projected/kube-api-access-qpmh6 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab539183-8f67-4c14-aef9-98b5924b6c33/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ab539183-8f67-4c14-aef9-98b5924b6c33/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~projected/kube-api-access-9m57w:{mountpoint:/var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~projected/kube-api-access-9m57w major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d70bada2-ec8a-40db-80d9-6455a9a675a3/volumes/kubernetes.io~projected/kube-api-access-69f2f:{mountpoint:/var/lib/kubelet/pods/d70bada2-ec8a-40db-80d9-6455a9a675a3/volumes/kubernetes.io~projected/kube-api-access-69f2f major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/volumes/kubernetes.io~projected/kube-api-access-jd8vt:{mountpoint:/var/lib/kubelet/pods/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/volumes/kubernetes.io~projected/kube-api-access-jd8vt major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2aab3d5-357d-470a-a27d-bab9d570a73d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e2aab3d5-357d-470a-a27d-bab9d570a73d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb3bbe34-cf5f-4d4f-88b9-600903ab0961/volumes/kubernetes.io~projected/kube-api-access-5hqmc:{mountpoint:/var/lib/kubelet/pods/eb3bbe34-cf5f-4d4f-88b9-600903ab0961/volumes/kubernetes.io~projected/kube-api-access-5hqmc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb3bbe34-cf5f-4d4f-88b9-600903ab0961/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/eb3bbe34-cf5f-4d4f-88b9-600903ab0961/volumes/kubernetes.io~secret/metrics-certs major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edcd5516-336e-415d-b587-c887774d8a63/volumes/kubernetes.io~projected/kube-api-access-fw8cx:{mountpoint:/var/lib/kubelet/pods/edcd5516-336e-415d-b587-c887774d8a63/volumes/kubernetes.io~projected/kube-api-access-fw8cx major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edcd5516-336e-415d-b587-c887774d8a63/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/edcd5516-336e-415d-b587-c887774d8a63/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edcd5516-336e-415d-b587-c887774d8a63/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/edcd5516-336e-415d-b587-c887774d8a63/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6bc998d-830b-4bed-9536-4e9a6424afe3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f6bc998d-830b-4bed-9536-4e9a6424afe3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6bc998d-830b-4bed-9536-4e9a6424afe3/volumes/kubernetes.io~projected/kube-api-access-7grjf:{mountpoint:/var/lib/kubelet/pods/f6bc998d-830b-4bed-9536-4e9a6424afe3/volumes/kubernetes.io~projected/kube-api-access-7grjf major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6bc998d-830b-4bed-9536-4e9a6424afe3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f6bc998d-830b-4bed-9536-4e9a6424afe3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a5a35f0efd62323bd7f1dc7cff5da841d3031989b4d241434975c55c0aa6eada/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e466aef87f2e3ee85dc650145e7f9f5889e871d4b88a389bd98181e16b9776f7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cc91ce6a032229184256e94c24651907e0c2615fc603965c2a1f374bc4858c48/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/96b9b9560d6f4050905649668623337fb7a4336dc10f1e456a1299a8b4bdbb2f/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/589ca3462f52c2c40ca7b0999661ab95bdacf984502a48b736cba5f4b9802ed6/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c74cb2b375333ad75f919713678b018e21bf044d739a018121d8afaccb02a596/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/84af0ffbb209cfdb6d572e2d68072b7f2414a7e89dd9d092c264f4bbfd8d1b26/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b42831ac5cf121f1dc3d71e47fee4a68a84bb0b491dfdc4095f87c43050f66a0/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e9b8f8717a1976be5a1781171da0083378253cfd9e497c717e0490ab7c9a76ae/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/dc68f74f4b30cbce6abdd87635427c1df5fac3b3df44d460bfd5564727ebb12e/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/aa17bd52ba0e206fb4ee7dbdf1171e38da7a76bf23a0535d7b2a934ea65dcfc4/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/887d59598444a33939840ffd4b970a112112be3840776317f074d6b72dcccbc6/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0a47736e37ce6b3424516afa07b3af9c40fe3bbaeeca33e8817d840460476bd0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/967001317e80223931edfa94ebaee1c901b104126c50782070895536a9089044/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9dd40a7b54939c00d1b48b2e2a8187823a385403f484d138b3e4e9b468c16de6/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0be7c5c3cc5824bbbf2088424ca83eafada0f0dffe53bae6d086ee1a86693bde/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/08f84de911e59bdc4e48a58216bb283f033aa833f73bc156f1d84c561238e0b8/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-173:{mountpoint:/var/lib/containers/storage/overlay/2c424103a84b0cf312a87b6860caef2281192498bba59bf3b96f725987f82da7/merged major:0 minor:173 fsType:overlay blockSize:0} overlay_0-175:{mountpoint:/var/lib/containers/storage/overlay/411366d49b22a1ba67bcc32c00c92d07198eb6bfbad8755bca0bcffe96abd73a/merged major:0 minor:175 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/02ab485b53f25677a64e5901ef7aaebc6235b37eaa4a613fd560e9829f6357a1/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0ef9c914bf2b8177fe41e7cb390f9770176f317ce4faf2a72cd575b5f8ddce61/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/5beebeec62f311e3fc00be7b64a2edbdce453467809c471546772202cacdbf2a/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/93f10434f0e6f75582126474e2cee0b715d1cf169b855325d42d2e9883157dc5/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/92d42b5036ebbdc161cc5bdec96e92de5284555df0f97908952c5ad5ee9f1049/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/0e53ddb77c2c1723a59dd2e70e9b77109ab0529558804db983095801d179e5ea/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/65b7f8af41ca100d0c003e4388c4b7ac0e9f2a2fcb092cfd76202a75b67e90d0/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/5d7770ee10bb45c7b6edd13b5d4d46bc69885bb9c4b1d3fdb1b9b869959878cc/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/ed84f2d439e52b0e487d33aadc05087f4d1d281c1f8349918477a13ebebc408a/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/d534a2369c51aa7b0b55f396bffe7963092d649a6e104eb24c5cc64e30d368df/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/d979a11b68f379af02c829ffdc0ecc25907f0154dcdd726c69b12710316d265e/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/c7dfe350f08135f3f31d113e2690228e6fe4e3638436386be6dfcfeb833d75d0/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/e23db2d38c13552a2532a5c42983aee5778ec47cdbde8d3f86f51a3d425ca802/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/4d9d87ab91012d3e7b6f8bcd7d878ef916c1fcd5f25058c72fa91095c343c62b/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/dce6e8d6be6a499dafb7847c74575cf51e936c07bd29ffbea9a814d12bd258f7/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/45598478bf055a81fa49c40a0b28860db1e9ba2aa92402bc86683bd1997dcbe6/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/7673ffe5df92c1efd172d2c32703bcbd3947b34eb80ecfd21c3b2fd2e7eda804/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/57ce70b96a3a74803de4df238fbf9b129c38f3dd05ab76ee6e99864921279e8d/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/1cf017fd2f8f3572e6a80c794e7940d0edbf2802993282594fd11b99782ff22f/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/792066224c7e434a4c2594cebd09731c5d25fe7cf1238af40ff70037727dae0b/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/50b846b76508404b5a30b1e387cc7cc3eddae2be30dad9a11db737dba789e24c/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/9c67a47312e21123d053bb8ec78a9484fcb3b658dfa8de506b77d9af6fea1c6e/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/d50d262ae1bb3f76bc786939fb8071aa2b1abbb7a8955b1c116440cfad9cab5a/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/91c62a91839116d18f0ed86fd7012c33507e36a73bcbbce0f890e02671da86dc/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/c127c594c7b9320a91093a42d17a5531bfadc28c96765fdd863edce5f3e05e0f/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/35a350d011a3ee49cc0b2783ff2670d3482d6a6b83e9f84e0f6c844832530cdd/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/48bde899e65de06adb1b218566017ba62705fa054db08a09a0436a94665cdfbe/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/c84316d33545ff761b088dfb4f25c06b4133dfa9a171c03fd4eb9e2e377fba32/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/536a6ed07839336baa8615767039997e2efd0d5978ce90cb2d9b63cda7a9b277/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/4f6fa05e9114d969262faf4c12e538969cac2a874ce3ff4c6a192717dabde0bc/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/69c99438c791f05a90d77c134ea58dfbc48edfed0da3c2f90e4990b516ab2a56/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/b52b739feab73dab20abd40b76002cd1b8210ca5f4d260697fd5f0c8441eb540/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/5c380aad90a0168a59fded2b1a25f5096661231fb9500fcc9db621e015f98623/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/c524752cc26bfcd56e0247c2bc7dcb13e8752e59399b7f962d885fd09b3e3d67/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/c73bb65dce54b3bcd720956fab0573f70eef3b4b31c3f18fbc1e920070eb1eaa/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/12a2c426afb36a802f8319e8327046cd00c11fef0deba6a4a2192ccf60fab7e8/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/18dfaf3de883bff8c31fe7dd029ee97cd938f7e9762627119ccf07ee65d58d28/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/bee8092a2600d600c1197b967f9c757b2925919907f8aae931e368d4a8d31d9c/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/6777562826e52a5989ff01d1328b690bea4f51114dd449916e3f29baed191eae/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/294c8e17ff8a542c92a76da841afcb06f833c05a00f11c2ec299ef05c2be6821/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/4a90b4ba0998f69812b47f24df15508525219144a5771acc8ede61a02d936c83/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/4699478b950ca741041e8766827eed3964c2e412a976f586730d7173f4a194ac/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/d44630073ba812d5be26719a6ac7a78ab4dc3b0a74385affa66ee3a6c306aea4/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/f5dafe9841f443a0ccfe5d27412c69aaa671b4256890be0efa0316a4748608b7/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/d986caaace8b6821930c8a8f6074a807df3b8d877e756b8e389d698889c1f699/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/63b21c4828a9bdccc87b13c3e20334af86fc8294fc640a22d5d01423d6517fb6/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/33be2182ebd5c4d959ef755a94df271c01e5c3e9d77de4075fcea80365e12b4e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/37ee1ae7b94c49431f960e956f66ba91b4146cda2be48776955420526dde0e1f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8f00d979d592ba9b36544ede1616e47752fc0b0f70e7936d8b76bcf7929d7ab9/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/365e95d193a05cde942fa3aa9dc77fb6b6ef112f6a5ed7720ceddbbc8542f63d/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a96490972093e98d7145ec1742e0e2a67a0bc200530b4cb05446a0953246c352/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a4d439951a80fc200eb47777543a61a3cd108f03056553fc692b198a0975da98/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/63742980ace1920115bd24a0b25b0754f85222cff57f466e43300a5f4c0d2b78/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/b1cf6d509c242a68db8be7e30a049a62f3b531b2f2e9dad2f465254d9410bbee/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9a130c7057fe9ac4665c1b09cabf9fea9bea4ba7b5b2ad3c772ce3e9d5e4aabe/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.548587    5600 manager.go:217] Machine: {Timestamp:2026-11-03 19:40:17.548080156 +0000 UTC m=+0.060901929 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100233 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2307ff6049715c009dd50d81b54870 SystemUUID:ec2307ff-6049-715c-009d-d50d81b54870 BootID:2f165dac-375f-40c2-b6b5-b64b115b9473 Filesystems:[{Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6bc998d-830b-4bed-9536-4e9a6424afe3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/2224bfc1-cc82-4e90-b723-e629d5f0f3f5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:268 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:376 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~projected/kube-api-access-tqrdl DeviceMajor:0 DeviceMinor:377 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39b2397af6c1565c258e80f46815d70fe4d820256d5b76c46112a390b842ade8/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/volumes/kubernetes.io~projected/kube-api-access-jd8vt DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:371 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-173 DeviceMajor:0 DeviceMinor:173 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7f7e72c047d1f63fa136e8bdf58d06b70d33d94913245861076f11e7eeadedb/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~projected/kube-api-access-s5qzh DeviceMajor:0 DeviceMinor:269 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48f72ab27ab404e9d978dfaf7e3d68a71d5db54d62bddf7f4cf8743da250db58/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:268 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:268 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf1d6b248740da36f5e293f6636bbf18e1c3da50d6144b5b437a9c702063cc51/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8bedff4e-9e2b-4582-bafe-29360cfa83ae/volumes/kubernetes.io~projected/kube-api-access-v8jnr DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa90567a31bf21c64a5651c62d660a07942d209f34ee252ff8a214d4e389da57/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:323 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55ed9d78bcef62604e945b7540a963036fb859a46533d9904b9c429d5d0b2048/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:265 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:376 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:372 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb232ad37c2c4153751d97b8e4c603375bdb7d55824e2f98d0005baec81a3471/userdata/shm DeviceMajor:0 DeviceMinor:380 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f6bc998d-830b-4bed-9536-4e9a6424afe3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:257 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:268 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:374 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:376 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edcd5516-336e-415d-b587-c887774d8a63/volumes/kubernetes.io~projected/kube-api-access-fw8cx DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:370 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce8887671e75021e0a061d1ded1eece177ef4a90b77b4a2d496fa15b1e9cb0c0/userdata/shm DeviceMajor:0 DeviceMinor:426 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:264 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:368 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16525d8b-c267-41c1-8048-cec8f8dc381f/volumes/kubernetes.io~projected/kube-api-access-88nc2 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:373 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:375 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/246af0f3d3dd0a114e9cdca6989f71416ae7e97c98ef154a92ddb038288644b6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d178b87-95d6-4985-bf0f-f0448dd784d7/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~projected/kube-api-access-9m57w DeviceMajor:0 DeviceMinor:330 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2aab3d5-357d-470a-a27d-bab9d570a73d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:420 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d70bada2-ec8a-40db-80d9-6455a9a675a3/volumes/kubernetes.io~projected/kube-api-access-69f2f DeviceMajor:0 DeviceMinor:423 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/81c1c0da-1ec1-4a75-9017-430bcedcc19b/volumes/kubernetes.io~projected/kube-api-access-4znh8 DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a2ca8b68-6873-4723-ad8f-92876443f8fa/volumes/kubernetes.io~projected/kube-api-access-qpmh6 DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc72b4ca04bc904a8d9e4c670bc021fe46d1623d6e1bdf0fb2b65e77b6736fdb/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27b11c5c6597d0250624a1141508bb56fd612ede51505a8053d7445197e92a6b/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bedff4e-9e2b-4582-bafe-29360cfa83ae/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:266 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c085aaf6c3736a7a8562e7defa661d2b376c6b508c5087b759117b2ab2ad6069/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:262 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:363 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:369 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/edcd5516-336e-415d-b587-c887774d8a63/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/5d178b87-95d6-4985-bf0f-f0448dd784d7/volumes/kubernetes.io~projected/kube-api-access-k6rp8 DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:376 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b69f39eef492a47cb2954c2dc9b5d18157ed2e092eccb392f9db0ab015b0f96/userdata/shm DeviceMajor:0 DeviceMinor:421 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6678bac546496281365a97a349a3cd41a1bc6a2055a4753a0ebb8f2dfe921f34/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eb3bbe34-cf5f-4d4f-88b9-600903ab0961/volumes/kubernetes.io~projected/kube-api-access-5hqmc DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:265 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab539183-8f67-4c14-aef9-98b5924b6c33/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b6f018189450cd956b8b25de13d8b2f5c6c0f236fd7835b69315c4840131a8a/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb3bbe34-cf5f-4d4f-88b9-600903ab0961/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:416 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3313a9fa2af18dd113f4cd4b3572b019043aba71daa7c3420db469c62537dee/userdata/shm DeviceMajor:0 DeviceMinor:428 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25/volumes/kubernetes.io~projected/kube-api-access-kpp59 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6bc998d-830b-4bed-9536-4e9a6424afe3/volumes/kubernetes.io~projected/kube-api-access-7grjf DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21752a7a0138e2305b6882d40ed02003ee0660c837bbfa7a2c48283d107a22be/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8668d2b7-8bf0-4786-b96f-0ba9ab45472c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d178b87-95d6-4985-bf0f-f0448dd784d7/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2224bfc1-cc82-4e90-b723-e629d5f0f3f5/volumes/kubernetes.io~projected/kube-api-access-rqvvg DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c31a30a7-dcde-418e-8297-f5af07990cb0/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0034d15b-8b67-4e49-8abd-b834b164af4f/volumes/kubernetes.io~projected/kube-api-access-slxrz DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad6d6c671a0cc6d8d687fcf5cf5f30dd3f1661857d4d9bd1488991d60f18c718/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/edcd5516-336e-415d-b587-c887774d8a63/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5ae888670d147d2333357f46f183055d310272c3896cbf5baed60d208c08d6f/userdata/shm DeviceMajor:0 DeviceMinor:283 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:367 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd674b248c6165f5d22a07c5a067c2db45322c3d1a86dcf72083d284a8d16775/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ae954350a2552cfef72be13db2b78f3546858a6c9221b01e1a27213b3c791bd/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-175 DeviceMajor:0 DeviceMinor:175 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8668d2b7-8bf0-4786-b96f-0ba9ab45472c/volumes/kubernetes.io~projected/kube-api-access-bphw8 DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:261 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:263 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/576764d6-ba4a-416e-aac2-13e08c76680a/volumes/kubernetes.io~projected/kube-api-access-2h8jg DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2fde71af-248d-498d-b358-4db5a52a8cd0/volumes/kubernetes.io~projected/kube-api-access-qlxts DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b587f04ed3984d763957dd90913c8bc07819228a05c9e8749b16df29c9299ff3/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:267 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0b6f018189450cd MacAddress:be:09:ed:2e:08:ef Speed:10000 Mtu:8901} {Name:39b2397af6c1565 MacAddress:a6:d4:32:e9:e8:db Speed:10000 Mtu:8901} {Name:55ed9d78bcef626 MacAddress:d6:65:90:9a:e5:ba Speed:10000 Mtu:8901} {Name:9b69f39eef492a4 MacAddress:0e:4e:ed:b7:28:eb Speed:10000 Mtu:8901} {Name:a5ae888670d147d MacAddress:12:fd:b3:94:26:4b Speed:10000 Mtu:8901} {Name:ad6d6c671a0cc6d MacAddress:3a:01:9a:09:09:9f Speed:10000 Mtu:8901} {Name:b3313a9fa2af18d MacAddress:0e:f2:28:98:b6:6c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cb:c4:98:6b Speed:0 Mtu:9001} {Name:br-int MacAddress:92:58:59:de:95:64 Speed:0 Mtu:8901} {Name:c085aaf6c3736a7 MacAddress:92:22:f3:f7:2a:25 Speed:10000 Mtu:8901} {Name:cb232ad37c2c415 MacAddress:9a:7e:e0:31:18:ce Speed:10000 Mtu:8901} {Name:ce8887671e75021 MacAddress:9e:be:e8:c2:03:17 Speed:10000 Mtu:8901} {Name:e7f7e72c047d1f6 MacAddress:02:5b:87:f4:5d:34 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cb:c4:98:6b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:76:30:74:b9:6f:7a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:73:40:66:a4:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.548994    5600 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549040    5600 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549329    5600 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549350    5600 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-24.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549486    5600 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549493    5600 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549504    5600 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549522    5600 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549600    5600 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549754    5600 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549853    5600 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549887    5600 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549901    5600 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549910    5600 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549918    5600 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:40:17.549990 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.549926    5600 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:40:17.550830 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.550576    5600 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:40:17.550830 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.550586    5600 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:40:17.551067 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.551052    5600 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:40:17.553615 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.553602    5600 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:40:17.553615 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.553615    5600 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:40:17.553726 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.553716    5600 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 19:40:17.553907 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.553896    5600 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:40:17.553943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.553926    5600 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:40:17.554041 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554032    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:40:17.554074 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554047    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:40:17.554074 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554053    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:40:17.554074 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554059    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:40:17.554074 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554069    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:40:17.554178 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554079    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:40:17.554178 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554094    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:40:17.554178 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554100    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:40:17.554178 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554106    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:40:17.554178 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554113    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:40:17.554178 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554121    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:40:17.554178 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554130    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:40:17.554178 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554156    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:40:17.554178 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.554166    5600 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:40:17.557905 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.557889    5600 server.go:1263] "Started kubelet"
Nov 03 19:40:17.558009 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.557959    5600 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:40:17.559310 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.558270    5600 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:40:17.559310 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.558354    5600 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:40:17.559310 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.558701    5600 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:40:17.558484 ip-10-0-128-24 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:40:17.559646 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.559333    5600 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:40:17.566460 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.562296    5600 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:40:17.567383 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.567359    5600 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:40:17.568762 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.567701    5600 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:40:17.568762 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.567721    5600 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:40:17.568762 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.567986    5600 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:40:17.568762 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:17.568536    5600 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-24.us-east-2.compute.internal\" not found"
Nov 03 19:40:17.568762 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.568543    5600 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:34:08 +0000 UTC" deadline="2028-04-26 05:18:46.075751655 +0000 UTC"
Nov 03 19:40:17.568762 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.568570    5600 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12945h38m28.507185296s"
Nov 03 19:40:17.570868 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.570846    5600 factory.go:55] Registering systemd factory
Nov 03 19:40:17.570868 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.570868    5600 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:40:17.571618 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.571599    5600 factory.go:153] Registering CRI-O factory
Nov 03 19:40:17.571618 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.571620    5600 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:40:17.571759 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.571678    5600 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:40:17.571759 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.571712    5600 factory.go:103] Registering Raw factory
Nov 03 19:40:17.571759 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.571728    5600 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:40:17.572183 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572146    5600 manager.go:319] Starting recovery of all containers
Nov 03 19:40:17.572399 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572371    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8ed709-4fdf-40b1-99c6-63e0d2a20e25" volumeName="kubernetes.io/projected/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-kube-api-access-kpp59" seLinuxMountContext=""
Nov 03 19:40:17.572513 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572405    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-config-out" seLinuxMountContext=""
Nov 03 19:40:17.572513 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572422    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ca8b68-6873-4723-ad8f-92876443f8fa" volumeName="kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:40:17.572513 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572474    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1" volumeName="kubernetes.io/configmap/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:40:17.572513 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572488    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1" volumeName="kubernetes.io/projected/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-kube-api-access-jd8vt" seLinuxMountContext=""
Nov 03 19:40:17.572513 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572501    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edcd5516-336e-415d-b587-c887774d8a63" volumeName="kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-tls" seLinuxMountContext=""
Nov 03 19:40:17.572513 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572513    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572554    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8ed709-4fdf-40b1-99c6-63e0d2a20e25" volumeName="kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-tmp" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572574    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572586    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-volume" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572620    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572635    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-db" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572647    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-kube-api-access-tqrdl" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572659    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ca8b68-6873-4723-ad8f-92876443f8fa" volumeName="kubernetes.io/projected/a2ca8b68-6873-4723-ad8f-92876443f8fa-kube-api-access-qpmh6" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572693    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16525d8b-c267-41c1-8048-cec8f8dc381f" volumeName="kubernetes.io/projected/16525d8b-c267-41c1-8048-cec8f8dc381f-kube-api-access-88nc2" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572704    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fde71af-248d-498d-b358-4db5a52a8cd0" volumeName="kubernetes.io/projected/2fde71af-248d-498d-b358-4db5a52a8cd0-kube-api-access-qlxts" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572716    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-tls-assets" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572728    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-config" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572739    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c31a30a7-dcde-418e-8297-f5af07990cb0" volumeName="kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572751    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c31a30a7-dcde-418e-8297-f5af07990cb0" volumeName="kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572763    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c31a30a7-dcde-418e-8297-f5af07990cb0" volumeName="kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572775    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d70bada2-ec8a-40db-80d9-6455a9a675a3" volumeName="kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572787    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d178b87-95d6-4985-bf0f-f0448dd784d7" volumeName="kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572798    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8668d2b7-8bf0-4786-b96f-0ba9ab45472c" volumeName="kubernetes.io/configmap/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-config-volume" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572810    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572822    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-metrics-client-certs" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572834    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6bc998d-830b-4bed-9536-4e9a6424afe3" volumeName="kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-env-overrides" seLinuxMountContext=""
Nov 03 19:40:17.572841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572846    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6bc998d-830b-4bed-9536-4e9a6424afe3" volumeName="kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572864    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-cluster-tls-config" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572877    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fde71af-248d-498d-b358-4db5a52a8cd0" volumeName="kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572951    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bedff4e-9e2b-4582-bafe-29360cfa83ae" volumeName="kubernetes.io/secret/8bedff4e-9e2b-4582-bafe-29360cfa83ae-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.572989    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edcd5516-336e-415d-b587-c887774d8a63" volumeName="kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573005    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6bc998d-830b-4bed-9536-4e9a6424afe3" volumeName="kubernetes.io/secret/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573034    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="576764d6-ba4a-416e-aac2-13e08c76680a" volumeName="kubernetes.io/projected/576764d6-ba4a-416e-aac2-13e08c76680a-kube-api-access-2h8jg" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573047    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573058    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-out" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573072    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2224bfc1-cc82-4e90-b723-e629d5f0f3f5" volumeName="kubernetes.io/projected/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-kube-api-access-rqvvg" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573084    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fde71af-248d-498d-b358-4db5a52a8cd0" volumeName="kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573097    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d178b87-95d6-4985-bf0f-f0448dd784d7" volumeName="kubernetes.io/projected/5d178b87-95d6-4985-bf0f-f0448dd784d7-kube-api-access-k6rp8" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573109    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c31a30a7-dcde-418e-8297-f5af07990cb0" volumeName="kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573121    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0034d15b-8b67-4e49-8abd-b834b164af4f" volumeName="kubernetes.io/configmap/0034d15b-8b67-4e49-8abd-b834b164af4f-serviceca" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573134    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573147    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573159    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-web-config" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573187    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c31a30a7-dcde-418e-8297-f5af07990cb0" volumeName="kubernetes.io/projected/c31a30a7-dcde-418e-8297-f5af07990cb0-kube-api-access-9m57w" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573199    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1" volumeName="kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573211    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1" volumeName="kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573223    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb3bbe34-cf5f-4d4f-88b9-600903ab0961" volumeName="kubernetes.io/projected/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-kube-api-access-5hqmc" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573234    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573247    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edcd5516-336e-415d-b587-c887774d8a63" volumeName="kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573259    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edcd5516-336e-415d-b587-c887774d8a63" volumeName="kubernetes.io/projected/edcd5516-336e-415d-b587-c887774d8a63-kube-api-access-fw8cx" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573271    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edcd5516-336e-415d-b587-c887774d8a63" volumeName="kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573304    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-tls-assets" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573315    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d178b87-95d6-4985-bf0f-f0448dd784d7" volumeName="kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573327    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8668d2b7-8bf0-4786-b96f-0ba9ab45472c" volumeName="kubernetes.io/secret/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-metrics-tls" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573338    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573377    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c31a30a7-dcde-418e-8297-f5af07990cb0" volumeName="kubernetes.io/empty-dir/c31a30a7-dcde-418e-8297-f5af07990cb0-audit-log" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573389    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2aab3d5-357d-470a-a27d-bab9d570a73d" volumeName="kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573403    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edcd5516-336e-415d-b587-c887774d8a63" volumeName="kubernetes.io/empty-dir/edcd5516-336e-415d-b587-c887774d8a63-volume-directive-shadow" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573424    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573461    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573472    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-grpc-tls" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573486    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6bc998d-830b-4bed-9536-4e9a6424afe3" volumeName="kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-config" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573497    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573510    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8668d2b7-8bf0-4786-b96f-0ba9ab45472c" volumeName="kubernetes.io/projected/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-kube-api-access-bphw8" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573523    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bedff4e-9e2b-4582-bafe-29360cfa83ae" volumeName="kubernetes.io/empty-dir/8bedff4e-9e2b-4582-bafe-29360cfa83ae-data-volume" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573534    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bedff4e-9e2b-4582-bafe-29360cfa83ae" volumeName="kubernetes.io/projected/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-api-access-v8jnr" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573545    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-kube-api-access-s5qzh" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573564    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="576764d6-ba4a-416e-aac2-13e08c76680a" volumeName="kubernetes.io/configmap/576764d6-ba4a-416e-aac2-13e08c76680a-iptables-alerter-script" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573575    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d178b87-95d6-4985-bf0f-f0448dd784d7" volumeName="kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573587    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573599    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573610    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab539183-8f67-4c14-aef9-98b5924b6c33" volumeName="kubernetes.io/secret/ab539183-8f67-4c14-aef9-98b5924b6c33-agent-certs" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573621    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c31a30a7-dcde-418e-8297-f5af07990cb0" volumeName="kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-client-ca-bundle" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573637    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0034d15b-8b67-4e49-8abd-b834b164af4f" volumeName="kubernetes.io/projected/0034d15b-8b67-4e49-8abd-b834b164af4f-kube-api-access-slxrz" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573652    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ca8b68-6873-4723-ad8f-92876443f8fa" volumeName="kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-daemon-config" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573664    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fde71af-248d-498d-b358-4db5a52a8cd0" volumeName="kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573676    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2224bfc1-cc82-4e90-b723-e629d5f0f3f5" volumeName="kubernetes.io/secret/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-cert" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573689    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bedff4e-9e2b-4582-bafe-29360cfa83ae" volumeName="kubernetes.io/configmap/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573702    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6bc998d-830b-4bed-9536-4e9a6424afe3" volumeName="kubernetes.io/projected/f6bc998d-830b-4bed-9536-4e9a6424afe3-kube-api-access-7grjf" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573715    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-web-config" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573727    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d178b87-95d6-4985-bf0f-f0448dd784d7" volumeName="kubernetes.io/empty-dir/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-textfile" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573738    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573751    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb3bbe34-cf5f-4d4f-88b9-600903ab0961" volumeName="kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573764    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573776    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573790    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8ed709-4fdf-40b1-99c6-63e0d2a20e25" volumeName="kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-tuned" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573801    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d178b87-95d6-4985-bf0f-f0448dd784d7" volumeName="kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-tls" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573813    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81c1c0da-1ec1-4a75-9017-430bcedcc19b" volumeName="kubernetes.io/projected/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kube-api-access-4znh8" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573824    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8668d2b7-8bf0-4786-b96f-0ba9ab45472c" volumeName="kubernetes.io/empty-dir/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-tmp-dir" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573838    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d7d54f2-9efe-467f-82a4-693e30e17b58" volumeName="kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573849    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab539183-8f67-4c14-aef9-98b5924b6c33" volumeName="kubernetes.io/configmap/ab539183-8f67-4c14-aef9-98b5924b6c33-konnectivity-ca" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573861    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16525d8b-c267-41c1-8048-cec8f8dc381f" volumeName="kubernetes.io/empty-dir/16525d8b-c267-41c1-8048-cec8f8dc381f-tmp-dir" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573873    5600 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fd1ca4f-1929-48df-8769-b2e469863d7a" volumeName="kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-main-db" seLinuxMountContext=""
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573909    5600 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:40:17.573946 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573918    5600 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:40:17.577468 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.573625    5600 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:40:17.577468 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.577378    5600 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:40:17.577754 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.577734    5600 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:40:17.578155 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.578135    5600 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 19:40:17.604495 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.604460    5600 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:40:17.608608 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.608588    5600 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:40:17.608608 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.608608    5600 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:40:17.608776 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.608627    5600 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:40:17.608776 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:17.608667    5600 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 19:40:17.612333 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.612089    5600 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:40:17.621927 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.621895    5600 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="8e440fc30202cf5b1daedc53f0f497b1447bc3f03e91956bb14a365e2a1a5e22" exitCode=0
Nov 03 19:40:17.621927 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.621926    5600 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="b91debb8fa0b8f21db59d794bb9847bc9d07316edd68d60b22711705b739a3e4" exitCode=0
Nov 03 19:40:17.622088 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.621937    5600 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="26f95f8c7c2d0697449a44a4d927a4c71f356b74e024ff3ccb3e67efb4abcaa5" exitCode=0
Nov 03 19:40:17.622088 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.621943    5600 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="cb3878b75c9491860cebbd7701f6561c580c1ed809d2dc5004945fdc92d4701b" exitCode=0
Nov 03 19:40:17.622088 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.621948    5600 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="ba0b89d577fa63e9ff7db151f4c420aefdf254ce886b2cdf49ce90588c7c8f92" exitCode=0
Nov 03 19:40:17.622088 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.621953    5600 generic.go:356] "Generic (PLEG): container finished" podID="2fde71af-248d-498d-b358-4db5a52a8cd0" containerID="57f96087f383ab4daa9314e99aca14641a157f8c49b8cb1b3510a8756cd142c7" exitCode=0
Nov 03 19:40:17.629423 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.629393    5600 generic.go:356] "Generic (PLEG): container finished" podID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerID="c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a" exitCode=0
Nov 03 19:40:17.644865 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.644810    5600 generic.go:356] "Generic (PLEG): container finished" podID="dd5aff924ba523805f662a53447cba5d" containerID="a8cdb9cc8396d06ff4385b752eb04121cf98a4edd91fa856c4813a4cb940d70d" exitCode=0
Nov 03 19:40:17.652825 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.652801    5600 generic.go:356] "Generic (PLEG): container finished" podID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerID="e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f" exitCode=0
Nov 03 19:40:17.657409 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.657388    5600 generic.go:356] "Generic (PLEG): container finished" podID="5d178b87-95d6-4985-bf0f-f0448dd784d7" containerID="caf17b9b146ebca4fe9bdb860350e02b1a8041f319968de3e5adbc608df1e052" exitCode=0
Nov 03 19:40:17.684118 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.684052    5600 manager.go:324] Recovery completed
Nov 03 19:40:17.708624 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.708594    5600 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:40:17.708624 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.708615    5600 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:40:17.708824 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.708639    5600 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:40:17.708824 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:17.708708    5600 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 19:40:17.708824 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.708796    5600 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 19:40:17.708976 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.708811    5600 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 19:40:17.708976 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.708933    5600 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 19:40:17.708976 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.708944    5600 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 19:40:17.708976 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.708952    5600 policy_none.go:50] "Start"
Nov 03 19:40:17.708976 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.708968    5600 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:40:17.709193 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.708981    5600 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:40:17.709193 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.709165    5600 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 19:40:17.709193 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.709176    5600 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 19:40:17.709193 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.709184    5600 policy_none.go:44] "Start"
Nov 03 19:40:17.713851 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.713825    5600 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:40:17.716516 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.716500    5600 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:40:17.717055 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:17.716524    5600 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:40:17.717055 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.717054    5600 server.go:87] "Starting device plugin registration server"
Nov 03 19:40:17.717323 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.717310    5600 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:40:17.717395 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.717321    5600 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:40:17.717477 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.717458    5600 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:40:17.717726 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.717569    5600 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:40:17.717726 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.717620    5600 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:40:17.717726 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.717627    5600 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:40:17.717726 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.717664    5600 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:40:17.717602521Z","UUID":"36596a9e-a5b9-4295-b202-b8a71af8956a","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:40:17.719393 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:17.719375    5600 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:40:17.719506 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.719468    5600 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:40:17.719506 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.719493    5600 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:40:17.817476 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.817419    5600 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:40:17.819604 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.819575    5600 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:40:17.819732 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.819616    5600 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:40:17.819732 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.819631    5600 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-24.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:40:17.819732 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.819693    5600 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:17.833978 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.833952    5600 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:17.834108 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.834037    5600 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:17.909103 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.909043    5600 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"]
Nov 03 19:40:17.909688 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.909642    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal" event={"ID":"92c0a5455fd7c0c77f98e0c6b3176e58","Type":"ContainerStarted","Data":"8c91e656d32f823c7b8102c1099bead51d014fc5f3ca94ac37cc540a1947e1e8"}
Nov 03 19:40:17.909688 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.909680    5600 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:17.909688 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.909687    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal" event={"ID":"92c0a5455fd7c0c77f98e0c6b3176e58","Type":"ContainerStarted","Data":"6678bac546496281365a97a349a3cd41a1bc6a2055a4753a0ebb8f2dfe921f34"}
Nov 03 19:40:17.909915 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.909722    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal" event={"ID":"dd5aff924ba523805f662a53447cba5d","Type":"ContainerStarted","Data":"e7df215cddae7f6bb2d8bc2d724df0585817e0ab684301bc12a7291ef9dfae58"}
Nov 03 19:40:17.909915 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.909740    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal" event={"ID":"dd5aff924ba523805f662a53447cba5d","Type":"ContainerDied","Data":"a8cdb9cc8396d06ff4385b752eb04121cf98a4edd91fa856c4813a4cb940d70d"}
Nov 03 19:40:17.909915 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.909758    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal" event={"ID":"dd5aff924ba523805f662a53447cba5d","Type":"ContainerStarted","Data":"246af0f3d3dd0a114e9cdca6989f71416ae7e97c98ef154a92ddb038288644b6"}
Nov 03 19:40:17.909915 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.909900    5600 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:17.919175 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.919160    5600 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:40:17.919225 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:17.919201    5600 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:17.919719 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.919707    5600 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:40:17.919750 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:17.919733    5600 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:17.979099 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.979032    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92c0a5455fd7c0c77f98e0c6b3176e58-config\") pod \"kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"92c0a5455fd7c0c77f98e0c6b3176e58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:17.979099 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.979067    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:17.979211 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:17.979113    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:18.080154 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.080121    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:18.080154 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.080155    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:18.080312 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.080173    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92c0a5455fd7c0c77f98e0c6b3176e58-config\") pod \"kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"92c0a5455fd7c0c77f98e0c6b3176e58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:18.080312 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.080218    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:18.080312 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.080227    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd5aff924ba523805f662a53447cba5d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"dd5aff924ba523805f662a53447cba5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:18.080312 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.080258    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92c0a5455fd7c0c77f98e0c6b3176e58-config\") pod \"kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal\" (UID: \"92c0a5455fd7c0c77f98e0c6b3176e58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:18.550866 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.550833    5600 apiserver.go:52] "Watching apiserver"
Nov 03 19:40:18.564110 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.564089    5600 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:40:18.564871 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.564851    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0","openshift-dns/node-resolver-zbfhf","openshift-monitoring/prometheus-k8s-0","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal","openshift-monitoring/node-exporter-zftbb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc","openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d","openshift-multus/multus-additional-cni-plugins-t8qlt","openshift-insights/insights-runtime-extractor-btkpj","openshift-monitoring/metrics-server-cd7f87d47-wq4k7","openshift-network-diagnostics/network-check-target-9bwkj","kube-system/global-pull-secret-syncer-9sbm7","openshift-multus/multus-hqlmf","openshift-cluster-node-tuning-operator/tuned-txdfv","openshift-multus/network-metrics-daemon-mr2m2","kube-system/konnectivity-agent-x8dsr","openshift-dns/dns-default-zgwk6","openshift-network-operator/iptables-alerter-4896h","openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7","openshift-image-registry/node-ca-hrqf7","openshift-ovn-kubernetes/ovnkube-node-ct5n8","kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal","openshift-ingress-canary/ingress-canary-62g8g"]
Nov 03 19:40:18.570197 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.570155    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zcfm6\""
Nov 03 19:40:18.571061 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571028    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:40:18.571061 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571040    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:40:18.571231 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571078    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:40:18.571231 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571135    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:40:18.571231 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571159    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:40:18.571231 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571184    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:40:18.571231 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571204    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:40:18.571231 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571207    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:40:18.571770 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571754    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:40:18.571849 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571823    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zxqwg\""
Nov 03 19:40:18.571900 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571877    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:40:18.571949 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.571921    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rlc2f\""
Nov 03 19:40:18.572035 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.572020    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rq9xl\""
Nov 03 19:40:18.572202 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.572190    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:40:18.572842 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.572822    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d4wgq\""
Nov 03 19:40:18.573729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.573677    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:40:18.573729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.573697    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:40:18.573847 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.573739    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:40:18.573847 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.573817    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:40:18.573903 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.573885    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:40:18.574367 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.574347    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:40:18.574425 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.574389    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7xvgk\""
Nov 03 19:40:18.575039 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.575026    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:40:18.575238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.575222    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:40:18.575281 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.575222    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:40:18.575281 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.575231    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:40:18.575593 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.575573    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:40:18.575702 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.575610    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f2rb8\""
Nov 03 19:40:18.575702 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.575608    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:40:18.576512 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.576490    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:40:18.576600 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.576577    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5rk8\""
Nov 03 19:40:18.576653 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.576635    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-drbgz\""
Nov 03 19:40:18.576739 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.576720    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zzp2g\""
Nov 03 19:40:18.576959 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.576940    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:40:18.577049 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.577034    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vmwsk\""
Nov 03 19:40:18.577049 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.577044    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:40:18.577159 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.577034    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:40:18.577159 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.577068    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:40:18.577159 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.577120    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:40:18.578105 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.578073    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:40:18.578105 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.578100    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:40:18.578230 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.578137    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbmvx\""
Nov 03 19:40:18.578230 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.578156    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:40:18.578296 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.578269    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tlzjx\""
Nov 03 19:40:18.578368 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.578353    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:40:18.578477 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.578426    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:40:18.578477 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.578456    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:40:18.579052 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.579034    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:40:18.579395 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.579361    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:40:18.579395 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.579378    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:40:18.579705 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.579682    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:40:18.579802 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.579773    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:40:18.579802 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.579780    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 19:40:18.579904 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.579829    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hnqg\""
Nov 03 19:40:18.580045 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.580028    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mvxmh\""
Nov 03 19:40:18.580650 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.580630    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:40:18.580736 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.580720    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:40:18.580859 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.580837    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:40:18.580968 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.580953    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 19:40:18.581284 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581271    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 19:40:18.581362 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581344    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 19:40:18.581415 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581397    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vgtlv\""
Nov 03 19:40:18.581470 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581404    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:40:18.581470 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581419    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g7dvr\""
Nov 03 19:40:18.581470 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581461    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nbbcm\""
Nov 03 19:40:18.581583 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581503    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 19:40:18.581612 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581579    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:40:18.581612 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581600    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-etc-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.581667 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581615    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-cnibin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.581667 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581630    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.581667 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581647    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-config\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.581814 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581797    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-sys-fs\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.581850 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581837    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-systemd-units\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.581879 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581862    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-bin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.581912 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581888    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysconfig\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.581912 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581904    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-host\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.581974 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581928    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.581974 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581963    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovn-node-metrics-cert\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.581974 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581970    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-config\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.582077 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.581993    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-netns\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.582077 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.582023    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-multus\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.582077 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.582063    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-conf-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.582279 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.582120    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:40:18.582279 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.582131    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.582279 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.582136    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxts\" (UniqueName: \"kubernetes.io/projected/2fde71af-248d-498d-b358-4db5a52a8cd0-kube-api-access-qlxts\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.582279 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.582154    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:40:18.582279 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.582265    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovn-node-metrics-cert\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.582647 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.582628    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-netns\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.583419 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583395    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-var-lib-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.583516 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583465    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-bin\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.583516 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583501    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmh6\" (UniqueName: \"kubernetes.io/projected/a2ca8b68-6873-4723-ad8f-92876443f8fa-kube-api-access-qpmh6\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.583629 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583533    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-run\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.583629 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583566    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-system-cni-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.583629 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583618    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.583757 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583652    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.583757 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583682    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-conf\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.583757 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583712    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-script-lib\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.583757 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583742    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grjf\" (UniqueName: \"kubernetes.io/projected/f6bc998d-830b-4bed-9536-4e9a6424afe3-kube-api-access-7grjf\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.583923 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583773    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-registration-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.583923 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583805    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-modprobe-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.583923 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583833    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-sys\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.583923 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583856    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-netd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.583923 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583886    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-env-overrides\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.583923 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583917    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-socket-dir-parent\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.584168 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583946    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-kubelet\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.584168 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.583977    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-etc-kubernetes\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.584168 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584004    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-systemd\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.584168 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584027    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-cnibin\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.584168 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584055    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-kubelet\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.584168 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584083    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-node-log\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.584168 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584112    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-system-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.584168 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584136    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.584168 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584166    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-daemon-config\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584201    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-lib-modules\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584232    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584261    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-etc-selinux\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584289    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-ovn\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584320    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-k8s-cni-cncf-io\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584349    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-tmp\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584379    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584415    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-cni-binary-copy\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584466    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584497    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-slash\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584522    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.584560 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584554    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.585087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584594    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpp59\" (UniqueName: \"kubernetes.io/projected/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-kube-api-access-kpp59\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.585087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584624    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-socket-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.585087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584649    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-multus-certs\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.585087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584678    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-var-lib-kubelet\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.585087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584707    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-tuned\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.585087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584735    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-systemd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.585087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584765    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-log-socket\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.585087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584790    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-hostroot\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.585087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584819    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-kubernetes\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.585087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584847    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-device-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.585087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.584879    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znh8\" (UniqueName: \"kubernetes.io/projected/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kube-api-access-4znh8\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.585701 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.585678    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.585962 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.585943    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-ovnkube-script-lib\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.586403 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.586378    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-tuned\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.586545 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.586521    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-tmp\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.586654 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.586618    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fde71af-248d-498d-b358-4db5a52a8cd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.586721 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.586706    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-daemon-config\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.586772 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.586747    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2ca8b68-6873-4723-ad8f-92876443f8fa-cni-binary-copy\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.587121 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.587103    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6bc998d-830b-4bed-9536-4e9a6424afe3-env-overrides\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.587867 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.587839    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 19:40:18.587867 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.587852    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 19:40:18.587867 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.587860    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 19:40:18.588048 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.588033    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 19:40:18.588361 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.588342    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 19:40:18.588361 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.588351    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 19:40:18.590221 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.590208    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 19:40:18.590278 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.590237    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kcnh9\""
Nov 03 19:40:18.594392 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.594377    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 19:40:18.606619 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.606596    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69f2f\" (UniqueName: \"kubernetes.io/projected/d70bada2-ec8a-40db-80d9-6455a9a675a3-kube-api-access-69f2f\") pod \"network-check-target-9bwkj\" (UID: \"d70bada2-ec8a-40db-80d9-6455a9a675a3\") " pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:40:18.608848 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.608831    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpp59\" (UniqueName: \"kubernetes.io/projected/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-kube-api-access-kpp59\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.610524 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.610501    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxts\" (UniqueName: \"kubernetes.io/projected/2fde71af-248d-498d-b358-4db5a52a8cd0-kube-api-access-qlxts\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.610703 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.610688    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znh8\" (UniqueName: \"kubernetes.io/projected/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kube-api-access-4znh8\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.611015 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.610998    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grjf\" (UniqueName: \"kubernetes.io/projected/f6bc998d-830b-4bed-9536-4e9a6424afe3-kube-api-access-7grjf\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.612141 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.612125    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmh6\" (UniqueName: \"kubernetes.io/projected/a2ca8b68-6873-4723-ad8f-92876443f8fa-kube-api-access-qpmh6\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.669685 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.669655    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 19:40:18.669916 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.669728    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 19:40:18.669916 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.669753    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 19:40:18.669916 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.669772    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 19:40:18.669916 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.669795    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 19:40:18.670189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.669919    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 19:40:18.670189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.669978    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 19:40:18.670189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.669995    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 19:40:18.670189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.669993    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d5pj80fngeohf\""
Nov 03 19:40:18.670189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.670020    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 19:40:18.670189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.670040    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-67p1b6fhp1pgt\""
Nov 03 19:40:18.670978 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.670955    5600 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:40:18.671104 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.671081    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x7mbg\""
Nov 03 19:40:18.671189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.671115    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 19:40:18.671189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.671159    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 19:40:18.671189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.671158    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 19:40:18.671189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.671173    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2wh29\""
Nov 03 19:40:18.671189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.671159    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 19:40:18.671475 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.671278    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 19:40:18.671475 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.671351    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 19:40:18.671475 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.671371    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 19:40:18.678888 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.678867    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 19:40:18.680817 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.680800    5600 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:18.680945 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.680929    5600 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:18.682265 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.682239    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 19:40:18.682472 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.682453    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 19:40:18.685329 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685308    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-systemd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.685458 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685341    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-log-socket\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.685458 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685383    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.685458 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685408    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rp8\" (UniqueName: \"kubernetes.io/projected/5d178b87-95d6-4985-bf0f-f0448dd784d7-kube-api-access-k6rp8\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.685458 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685418    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-systemd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.685458 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685447    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-device-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685477    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685485    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-log-socket\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685504    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685535    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-device-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685527    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jnr\" (UniqueName: \"kubernetes.io/projected/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-api-access-v8jnr\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685575    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-etc-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685600    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685625    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-cnibin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685648    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-sys-fs\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685672    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-tls\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685708    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-systemd-units\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.685729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685730    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysconfig\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685737    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685758    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd8vt\" (UniqueName: \"kubernetes.io/projected/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-kube-api-access-jd8vt\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685769    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685793    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-netns\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685810    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685840    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-netns\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685682    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-etc-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685860    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-systemd-units\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685911    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-sys-fs\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685912    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-cnibin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685912    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysconfig\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685938    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685958    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-bin\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.685988    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-tls\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686024    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-bin\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686044    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686068    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686093    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8bedff4e-9e2b-4582-bafe-29360cfa83ae-crio-socket\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686141    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686174    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686189    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686195    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686203    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-modprobe-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.686215 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686219    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-sys\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686232    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-netd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686250    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88nc2\" (UniqueName: \"kubernetes.io/projected/16525d8b-c267-41c1-8048-cec8f8dc381f-kube-api-access-88nc2\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686271    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686286    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-sys\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686294    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-socket-dir-parent\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686320    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-etc-kubernetes\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686296    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-cni-netd\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686336    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-modprobe-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686343    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-systemd\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686352    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-socket-dir-parent\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686366    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-cnibin\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686319    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686352    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686370    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-etc-kubernetes\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686393    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-cnibin\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686400    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-systemd\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686406    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686418    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-node-log\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686459    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-node-log\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686461    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw8cx\" (UniqueName: \"kubernetes.io/projected/edcd5516-336e-415d-b587-c887774d8a63-kube-api-access-fw8cx\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686487    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-system-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686522    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-lib-modules\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686559    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686576    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-system-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686584    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0034d15b-8b67-4e49-8abd-b834b164af4f-serviceca\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686607    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686633    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-wtmp\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686654    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686660    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-lib-modules\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686657    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-web-config\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.687020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.686730    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687124    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687212    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687301    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-ovn\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687372    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-run-ovn\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687379    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-web-config\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687487    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687482    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0034d15b-8b67-4e49-8abd-b834b164af4f-serviceca\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687527    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687566    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687598    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-kubelet-config\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687616    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687633    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-tmp-dir\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687694    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687734    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-dbus\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687776    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8bedff4e-9e2b-4582-bafe-29360cfa83ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687810    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c31a30a7-dcde-418e-8297-f5af07990cb0-audit-log\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.687977    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.688489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.688230    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-metrics-certs\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:40:18.689256 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.688507    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.689256 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.688599    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-tmp-dir\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:40:18.689256 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.688880    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8bedff4e-9e2b-4582-bafe-29360cfa83ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:40:18.689256 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689044    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c31a30a7-dcde-418e-8297-f5af07990cb0-audit-log\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.689544 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689522    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-client-ca-bundle\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.689597 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689568    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.689690 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689599    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-cert\") pod \"ingress-canary-62g8g\" (UID: \"2224bfc1-cc82-4e90-b723-e629d5f0f3f5\") " pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:40:18.689690 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689630    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-tls\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.689690 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689661    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-socket-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.689690 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689686    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqmc\" (UniqueName: \"kubernetes.io/projected/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-kube-api-access-5hqmc\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:40:18.689855 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689717    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvvg\" (UniqueName: \"kubernetes.io/projected/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-kube-api-access-rqvvg\") pod \"ingress-canary-62g8g\" (UID: \"2224bfc1-cc82-4e90-b723-e629d5f0f3f5\") " pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:40:18.689855 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689746    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.689855 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689776    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.689855 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689805    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.689855 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689831    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.690055 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689860    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:40:18.690055 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689890    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab539183-8f67-4c14-aef9-98b5924b6c33-agent-certs\") pod \"konnectivity-agent-x8dsr\" (UID: \"ab539183-8f67-4c14-aef9-98b5924b6c33\") " pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:40:18.690055 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689918    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bphw8\" (UniqueName: \"kubernetes.io/projected/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-kube-api-access-bphw8\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:40:18.690055 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689946    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:40:18.690055 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.689977    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.690055 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690007    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-hostroot\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.690055 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690036    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-kubernetes\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.690350 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690067    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.690350 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690101    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bedff4e-9e2b-4582-bafe-29360cfa83ae-data-volume\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:40:18.690350 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690132    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m57w\" (UniqueName: \"kubernetes.io/projected/c31a30a7-dcde-418e-8297-f5af07990cb0-kube-api-access-9m57w\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.690350 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690163    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.690350 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690195    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0034d15b-8b67-4e49-8abd-b834b164af4f-host\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:40:18.690350 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690224    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.690350 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690274    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.690350 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690305    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.690350 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690340    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrdl\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-kube-api-access-tqrdl\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.690738 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690371    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/576764d6-ba4a-416e-aac2-13e08c76680a-iptables-alerter-script\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:40:18.690738 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.690398    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:40:18.691246 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691224    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-kubernetes\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.691618 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691348    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.691708 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691649    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-volume\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.691708 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691667    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-hostroot\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.691817 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691703    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.691817 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691726    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.691817 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691731    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bedff4e-9e2b-4582-bafe-29360cfa83ae-data-volume\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:40:18.691817 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691740    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-metrics-server-audit-profiles\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.691817 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691770    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.691817 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691777    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.692129 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691828    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-volume\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.692129 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691872    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab539183-8f67-4c14-aef9-98b5924b6c33-agent-certs\") pod \"konnectivity-agent-x8dsr\" (UID: \"ab539183-8f67-4c14-aef9-98b5924b6c33\") " pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:40:18.692129 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691878    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.692129 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691870    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.692129 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691906    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2aab3d5-357d-470a-a27d-bab9d570a73d-original-pull-secret\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:40:18.692129 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691913    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.692129 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.691955    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.692129 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692037    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.692529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692194    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.692529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692293    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.692529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692342    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c31a30a7-dcde-418e-8297-f5af07990cb0-metrics-server-audit-profiles\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.692529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692485    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-tls\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.692529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692528    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-bin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692536    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-bin\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692562    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-host\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692571    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-run-ovn-kubernetes\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692587    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692605    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692646    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-host\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692663    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692697    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-netns\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692730    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-multus\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692761    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692774    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.692787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692763    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-conf-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692805    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-conf-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692865    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/576764d6-ba4a-416e-aac2-13e08c76680a-iptables-alerter-script\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692874    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-netns\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692866    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-socket-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692947    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.692991    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-cni-multus\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693002    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693025    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-var-lib-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693051    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-config\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693074    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/576764d6-ba4a-416e-aac2-13e08c76680a-host-slash\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693089    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-run\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693088    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-var-lib-openvswitch\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693108    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-system-cni-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693120    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-cert\") pod \"ingress-canary-62g8g\" (UID: \"2224bfc1-cc82-4e90-b723-e629d5f0f3f5\") " pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693175    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693193    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fde71af-248d-498d-b358-4db5a52a8cd0-system-cni-dir\") pod \"multus-additional-cni-plugins-t8qlt\" (UID: \"2fde71af-248d-498d-b358-4db5a52a8cd0\") " pod="openshift-multus/multus-additional-cni-plugins-t8qlt"
Nov 03 19:40:18.693233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693200    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-out\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693242    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693260    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-out\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693278    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-conf\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693291    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-config\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693323    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-run\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693333    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-registration-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693358    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693379    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-sys\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693383    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-registration-dir\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693399    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-textfile\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693420    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slxrz\" (UniqueName: \"kubernetes.io/projected/0034d15b-8b67-4e49-8abd-b834b164af4f-kube-api-access-slxrz\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693450    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-conf\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693503    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qzh\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-kube-api-access-s5qzh\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693539    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693550    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693572    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-client-certs\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693611    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693617    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/edcd5516-336e-415d-b587-c887774d8a63-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693641    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693663    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693711    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-kubelet\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693736    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-kubelet\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693747    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693758    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693767    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-textfile\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693799    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-var-lib-kubelet\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693836    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-multus-cni-dir\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693837    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-etc-selinux\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693858    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-kubelet\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693883    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-k8s-cni-cncf-io\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693904    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16525d8b-c267-41c1-8048-cec8f8dc381f-hosts-file\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693912    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693925    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.693963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.693956    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8jg\" (UniqueName: \"kubernetes.io/projected/576764d6-ba4a-416e-aac2-13e08c76680a-kube-api-access-2h8jg\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694003    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694032    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-config-volume\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694036    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81c1c0da-1ec1-4a75-9017-430bcedcc19b-etc-selinux\") pod \"aws-ebs-csi-driver-node-w62zc\" (UID: \"81c1c0da-1ec1-4a75-9017-430bcedcc19b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w62zc"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694045    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-k8s-cni-cncf-io\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694039    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-secret-metrics-server-client-certs\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694064    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694173    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694192    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-etc-sysctl-d\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694231    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694296    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694308    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-slash\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694324    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-config-volume\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694343    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694368    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6bc998d-830b-4bed-9536-4e9a6424afe3-host-slash\") pod \"ovnkube-node-ct5n8\" (UID: \"f6bc998d-830b-4bed-9536-4e9a6424afe3\") " pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694383    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-root\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694446    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-metrics-tls\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694491    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694516    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/edcd5516-336e-415d-b587-c887774d8a63-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694543    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694564    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694568    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694583    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab539183-8f67-4c14-aef9-98b5924b6c33-konnectivity-ca\") pod \"konnectivity-agent-x8dsr\" (UID: \"ab539183-8f67-4c14-aef9-98b5924b6c33\") " pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694603    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/edcd5516-336e-415d-b587-c887774d8a63-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694608    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16525d8b-c267-41c1-8048-cec8f8dc381f-tmp-dir\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694655    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-metrics-client-ca\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694678    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-metrics-tls\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694704    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-multus-certs\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694751    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-var-lib-kubelet\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694768    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2ca8b68-6873-4723-ad8f-92876443f8fa-host-run-multus-certs\") pod \"multus-hqlmf\" (UID: \"a2ca8b68-6873-4723-ad8f-92876443f8fa\") " pod="openshift-multus/multus-hqlmf"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694771    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab539183-8f67-4c14-aef9-98b5924b6c33-konnectivity-ca\") pod \"konnectivity-agent-x8dsr\" (UID: \"ab539183-8f67-4c14-aef9-98b5924b6c33\") " pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694707    5600 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694800    5600 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:18.694838    5600 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:18.694846    5600 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-24.us-east-2.compute.internal"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694850    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694862    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f8ed709-4fdf-40b1-99c6-63e0d2a20e25-var-lib-kubelet\") pod \"tuned-txdfv\" (UID: \"4f8ed709-4fdf-40b1-99c6-63e0d2a20e25\") " pod="openshift-cluster-node-tuning-operator/tuned-txdfv"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.694932    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16525d8b-c267-41c1-8048-cec8f8dc381f-tmp-dir\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.695058    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.695150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.695159    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d178b87-95d6-4985-bf0f-f0448dd784d7-metrics-client-ca\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.696710 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.695991    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c31a30a7-dcde-418e-8297-f5af07990cb0-client-ca-bundle\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.696710 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.696462    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rp8\" (UniqueName: \"kubernetes.io/projected/5d178b87-95d6-4985-bf0f-f0448dd784d7-kube-api-access-k6rp8\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.696710 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.696656    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jnr\" (UniqueName: \"kubernetes.io/projected/8bedff4e-9e2b-4582-bafe-29360cfa83ae-kube-api-access-v8jnr\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:40:18.696922 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.696906    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw8cx\" (UniqueName: \"kubernetes.io/projected/edcd5516-336e-415d-b587-c887774d8a63-kube-api-access-fw8cx\") pod \"kube-state-metrics-5dc8d7d89b-rdgs7\" (UID: \"edcd5516-336e-415d-b587-c887774d8a63\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rdgs7"
Nov 03 19:40:18.697155 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.697134    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd8vt\" (UniqueName: \"kubernetes.io/projected/de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1-kube-api-access-jd8vt\") pod \"openshift-state-metrics-7bd7489957-qpx5d\" (UID: \"de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qpx5d"
Nov 03 19:40:18.697552 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.697535    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88nc2\" (UniqueName: \"kubernetes.io/projected/16525d8b-c267-41c1-8048-cec8f8dc381f-kube-api-access-88nc2\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:40:18.699639 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.699623    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvvg\" (UniqueName: \"kubernetes.io/projected/2224bfc1-cc82-4e90-b723-e629d5f0f3f5-kube-api-access-rqvvg\") pod \"ingress-canary-62g8g\" (UID: \"2224bfc1-cc82-4e90-b723-e629d5f0f3f5\") " pod="openshift-ingress-canary/ingress-canary-62g8g"
Nov 03 19:40:18.700998 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.700981    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qzh\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-kube-api-access-s5qzh\") pod \"alertmanager-main-0\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:18.701923 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.701907    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8jg\" (UniqueName: \"kubernetes.io/projected/576764d6-ba4a-416e-aac2-13e08c76680a-kube-api-access-2h8jg\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:40:18.702051 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.702030    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqmc\" (UniqueName: \"kubernetes.io/projected/eb3bbe34-cf5f-4d4f-88b9-600903ab0961-kube-api-access-5hqmc\") pod \"network-metrics-daemon-mr2m2\" (UID: \"eb3bbe34-cf5f-4d4f-88b9-600903ab0961\") " pod="openshift-multus/network-metrics-daemon-mr2m2"
Nov 03 19:40:18.702220 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.702202    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slxrz\" (UniqueName: \"kubernetes.io/projected/0034d15b-8b67-4e49-8abd-b834b164af4f-kube-api-access-slxrz\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:40:18.702460 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.702424    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphw8\" (UniqueName: \"kubernetes.io/projected/8668d2b7-8bf0-4786-b96f-0ba9ab45472c-kube-api-access-bphw8\") pod \"dns-default-zgwk6\" (UID: \"8668d2b7-8bf0-4786-b96f-0ba9ab45472c\") " pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:40:18.703043 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.703025    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m57w\" (UniqueName: \"kubernetes.io/projected/c31a30a7-dcde-418e-8297-f5af07990cb0-kube-api-access-9m57w\") pod \"metrics-server-cd7f87d47-wq4k7\" (UID: \"c31a30a7-dcde-418e-8297-f5af07990cb0\") " pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:18.703083 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.703059    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrdl\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-kube-api-access-tqrdl\") pod \"prometheus-k8s-0\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:18.795934 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.795901    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-sys\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.795934 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.795936    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16525d8b-c267-41c1-8048-cec8f8dc381f-hosts-file\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:40:18.796114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.795953    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-root\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.796114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.795978    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8bedff4e-9e2b-4582-bafe-29360cfa83ae-crio-socket\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:40:18.796114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796002    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-wtmp\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.796114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796008    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-sys\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.796114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796035    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16525d8b-c267-41c1-8048-cec8f8dc381f-hosts-file\") pod \"node-resolver-zbfhf\" (UID: \"16525d8b-c267-41c1-8048-cec8f8dc381f\") " pod="openshift-dns/node-resolver-zbfhf"
Nov 03 19:40:18.796114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796035    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-root\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.796114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796066    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8bedff4e-9e2b-4582-bafe-29360cfa83ae-crio-socket\") pod \"insights-runtime-extractor-btkpj\" (UID: \"8bedff4e-9e2b-4582-bafe-29360cfa83ae\") " pod="openshift-insights/insights-runtime-extractor-btkpj"
Nov 03 19:40:18.796114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796074    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d178b87-95d6-4985-bf0f-f0448dd784d7-node-exporter-wtmp\") pod \"node-exporter-zftbb\" (UID: \"5d178b87-95d6-4985-bf0f-f0448dd784d7\") " pod="openshift-monitoring/node-exporter-zftbb"
Nov 03 19:40:18.796114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796059    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-kubelet-config\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:40:18.796114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796101    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-kubelet-config\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:40:18.796114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796110    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-dbus\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:40:18.796524 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796137    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0034d15b-8b67-4e49-8abd-b834b164af4f-host\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:40:18.796524 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796144    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2aab3d5-357d-470a-a27d-bab9d570a73d-dbus\") pod \"global-pull-secret-syncer-9sbm7\" (UID: \"e2aab3d5-357d-470a-a27d-bab9d570a73d\") " pod="kube-system/global-pull-secret-syncer-9sbm7"
Nov 03 19:40:18.796524 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796163    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/576764d6-ba4a-416e-aac2-13e08c76680a-host-slash\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:40:18.796524 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796195    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0034d15b-8b67-4e49-8abd-b834b164af4f-host\") pod \"node-ca-hrqf7\" (UID: \"0034d15b-8b67-4e49-8abd-b834b164af4f\") " pod="openshift-image-registry/node-ca-hrqf7"
Nov 03 19:40:18.796524 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:18.796204    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/576764d6-ba4a-416e-aac2-13e08c76680a-host-slash\") pod \"iptables-alerter-4896h\" (UID: \"576764d6-ba4a-416e-aac2-13e08c76680a\") " pod="openshift-network-operator/iptables-alerter-4896h"
Nov 03 19:40:19.166467 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:19.166419    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zgwk6"
Nov 03 19:40:19.647279 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:19.647252    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:20.927148 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:20.927117    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:23.203504 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:23.203470    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:40:23.520497 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:23.520405    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:23.520639 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:23.520628    5600 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:40:25.595832 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:25.595801    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x8dsr"
Nov 03 19:40:28.794307 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:28.794224    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:40:33.885570 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:33.885539    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-cd7f87d47-wq4k7"
Nov 03 19:40:36.251493 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:36.251464    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ct5n8"
Nov 03 19:40:38.616891 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.616851    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:40:38.617717 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.617478    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="alertmanager" containerID="cri-o://e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5" gracePeriod=120
Nov 03 19:40:38.617717 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.617499    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="prom-label-proxy" containerID="cri-o://423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a" gracePeriod=120
Nov 03 19:40:38.617717 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.617558    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy-metric" containerID="cri-o://ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae" gracePeriod=120
Nov 03 19:40:38.617717 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.617566    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="config-reloader" containerID="cri-o://cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c" gracePeriod=120
Nov 03 19:40:38.617717 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.617507    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy-web" containerID="cri-o://d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9" gracePeriod=120
Nov 03 19:40:38.617717 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.617501    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy" containerID="cri-o://895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf" gracePeriod=120
Nov 03 19:40:38.757332 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.757306    5600 generic.go:356] "Generic (PLEG): container finished" podID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerID="423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a" exitCode=0
Nov 03 19:40:38.757332 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.757329    5600 generic.go:356] "Generic (PLEG): container finished" podID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerID="ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae" exitCode=0
Nov 03 19:40:38.757539 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.757337    5600 generic.go:356] "Generic (PLEG): container finished" podID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerID="895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf" exitCode=0
Nov 03 19:40:38.757539 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.757344    5600 generic.go:356] "Generic (PLEG): container finished" podID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerID="cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c" exitCode=0
Nov 03 19:40:38.757539 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.757351    5600 generic.go:356] "Generic (PLEG): container finished" podID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerID="e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5" exitCode=0
Nov 03 19:40:38.757539 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.757385    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerDied","Data":"423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a"}
Nov 03 19:40:38.757539 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.757489    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerDied","Data":"ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae"}
Nov 03 19:40:38.757539 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.757538    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerDied","Data":"895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf"}
Nov 03 19:40:38.757767 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.757577    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerDied","Data":"cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c"}
Nov 03 19:40:38.757767 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:38.757618    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerDied","Data":"e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5"}
Nov 03 19:40:39.963017 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:39.962995    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.086141 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086035    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-web\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.086141 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086101    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-out\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.086378 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086159    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-main-db\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.086378 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086190    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-cluster-tls-config\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.086378 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086219    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-kube-api-access-s5qzh\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-kube-api-access-s5qzh\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.086378 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086244    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-tls-assets\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-tls-assets\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.086378 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086269    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.086378 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086324    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-metrics-client-ca\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.086378 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086359    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.087131 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086388    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.087131 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086422    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-volume\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-volume\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.087131 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086483    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-main-tls\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.087131 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086485    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-main-db" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:40:40.087131 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086514    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-web-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-web-config\") pod \"1fd1ca4f-1929-48df-8769-b2e469863d7a\" (UID: \"1fd1ca4f-1929-48df-8769-b2e469863d7a\") "
Nov 03 19:40:40.087131 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086732    5600 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-main-db\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.087131 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.086756    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-metrics-client-ca" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:40.087687 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.087342    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:40.090024 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.089990    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:40.090024 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.090006    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-tls-assets" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:40:40.090388 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.090363    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-web" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:40.090500 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.090397    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-kube-api-access-s5qzh" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "kube-api-access-s5qzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:40:40.090729 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.090689    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-out" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:40:40.090891 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.090869    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-volume" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:40.090991 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.090976    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-main-tls" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:40.091068 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.091046    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-metric" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:40.094687 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.094585    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-cluster-tls-config" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:40.101637 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.101613    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-web-config" pod "1fd1ca4f-1929-48df-8769-b2e469863d7a" (UID: "1fd1ca4f-1929-48df-8769-b2e469863d7a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:40.187943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.187907    5600 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-cluster-tls-config\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.187943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.187935    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5qzh\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-kube-api-access-s5qzh\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.187943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.187946    5600 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1fd1ca4f-1929-48df-8769-b2e469863d7a-tls-assets\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.188180 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.187956    5600 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.188180 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.187967    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-metrics-client-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.188180 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.187976    5600 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fd1ca4f-1929-48df-8769-b2e469863d7a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.188180 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.187985    5600 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.188180 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.187995    5600 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-volume\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.188180 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.188003    5600 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-main-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.188180 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.188013    5600 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-web-config\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.188180 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.188022    5600 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1fd1ca4f-1929-48df-8769-b2e469863d7a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.188180 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.188030    5600 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1fd1ca4f-1929-48df-8769-b2e469863d7a-config-out\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:40.767290 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.767254    5600 generic.go:356] "Generic (PLEG): container finished" podID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerID="d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9" exitCode=0
Nov 03 19:40:40.767462 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.767300    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerDied","Data":"d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9"}
Nov 03 19:40:40.767462 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.767351    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.767462 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.767366    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fd1ca4f-1929-48df-8769-b2e469863d7a","Type":"ContainerDied","Data":"a5ae888670d147d2333357f46f183055d310272c3896cbf5baed60d208c08d6f"}
Nov 03 19:40:40.767462 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.767403    5600 scope.go:122] "RemoveContainer" containerID="423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a"
Nov 03 19:40:40.776862 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.776846    5600 scope.go:122] "RemoveContainer" containerID="ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae"
Nov 03 19:40:40.783921 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.783904    5600 scope.go:122] "RemoveContainer" containerID="895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf"
Nov 03 19:40:40.790385 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.790369    5600 scope.go:122] "RemoveContainer" containerID="d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9"
Nov 03 19:40:40.792217 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.792193    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:40:40.796497 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.796475    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:40:40.798716 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.798701    5600 scope.go:122] "RemoveContainer" containerID="cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c"
Nov 03 19:40:40.805324 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.805301    5600 scope.go:122] "RemoveContainer" containerID="e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5"
Nov 03 19:40:40.812016 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.812001    5600 scope.go:122] "RemoveContainer" containerID="c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a"
Nov 03 19:40:40.818911 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.818896    5600 scope.go:122] "RemoveContainer" containerID="423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a"
Nov 03 19:40:40.819207 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:40.819174    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a\": container with ID starting with 423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a not found: ID does not exist" containerID="423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a"
Nov 03 19:40:40.819280 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.819218    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a"} err="failed to get container status \"423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a\": rpc error: code = NotFound desc = could not find container \"423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a\": container with ID starting with 423788b4acff8c1efe6bca05104d7a6df4b84c37f66227dec74c99f732cf579a not found: ID does not exist"
Nov 03 19:40:40.819280 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.819251    5600 scope.go:122] "RemoveContainer" containerID="ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae"
Nov 03 19:40:40.819617 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:40.819597    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae\": container with ID starting with ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae not found: ID does not exist" containerID="ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae"
Nov 03 19:40:40.819700 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.819628    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae"} err="failed to get container status \"ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae\": rpc error: code = NotFound desc = could not find container \"ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae\": container with ID starting with ff8b4d60cfb24e13582349f71bb04a3259022f7124cfac93d446b7a267dc8cae not found: ID does not exist"
Nov 03 19:40:40.819700 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.819650    5600 scope.go:122] "RemoveContainer" containerID="895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf"
Nov 03 19:40:40.819920 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:40.819904    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf\": container with ID starting with 895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf not found: ID does not exist" containerID="895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf"
Nov 03 19:40:40.819966 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.819926    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf"} err="failed to get container status \"895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf\": rpc error: code = NotFound desc = could not find container \"895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf\": container with ID starting with 895747b2a137d8a6cccf3a6895d9d7805dbcd8ac2b88cf63b33772f8e4c023bf not found: ID does not exist"
Nov 03 19:40:40.819966 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.819941    5600 scope.go:122] "RemoveContainer" containerID="d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9"
Nov 03 19:40:40.820188 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:40.820170    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9\": container with ID starting with d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9 not found: ID does not exist" containerID="d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9"
Nov 03 19:40:40.820229 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.820194    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9"} err="failed to get container status \"d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9\": rpc error: code = NotFound desc = could not find container \"d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9\": container with ID starting with d99b60455983bb3b73f3730edf843cd8b664c18681524f031b646c00c076daa9 not found: ID does not exist"
Nov 03 19:40:40.820229 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.820207    5600 scope.go:122] "RemoveContainer" containerID="cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c"
Nov 03 19:40:40.820427 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:40.820411    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c\": container with ID starting with cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c not found: ID does not exist" containerID="cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c"
Nov 03 19:40:40.820509 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.820460    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c"} err="failed to get container status \"cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c\": rpc error: code = NotFound desc = could not find container \"cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c\": container with ID starting with cc231522527ad186559dfc5e48ab335c740d01a0d26a88cfab66a8567297c65c not found: ID does not exist"
Nov 03 19:40:40.820509 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.820480    5600 scope.go:122] "RemoveContainer" containerID="e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5"
Nov 03 19:40:40.820724 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:40.820710    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5\": container with ID starting with e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5 not found: ID does not exist" containerID="e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5"
Nov 03 19:40:40.820762 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.820728    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5"} err="failed to get container status \"e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5\": rpc error: code = NotFound desc = could not find container \"e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5\": container with ID starting with e82a960fb060b88b0bac5185f9b78bb579542b851204671b1e9b24e8b60b14e5 not found: ID does not exist"
Nov 03 19:40:40.820762 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.820741    5600 scope.go:122] "RemoveContainer" containerID="c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a"
Nov 03 19:40:40.820948 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:40.820932    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a\": container with ID starting with c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a not found: ID does not exist" containerID="c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a"
Nov 03 19:40:40.821000 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.820955    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a"} err="failed to get container status \"c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a\": rpc error: code = NotFound desc = could not find container \"c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a\": container with ID starting with c896ca284963d5fe5c1d714869f69ea9bcd491c81d8e9caa708b607bebcc646a not found: ID does not exist"
Nov 03 19:40:40.823422 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823384    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:40:40.823811 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823755    5600 cpu_manager.go:414] "containerMap: removing container" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy-metric"
Nov 03 19:40:40.823811 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823803    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy-metric"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823834    5600 cpu_manager.go:414] "containerMap: removing container" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823842    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823854    5600 cpu_manager.go:414] "containerMap: removing container" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="alertmanager"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823860    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="alertmanager"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823875    5600 cpu_manager.go:414] "containerMap: removing container" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="prom-label-proxy"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823882    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="prom-label-proxy"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823892    5600 cpu_manager.go:414] "containerMap: removing container" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="config-reloader"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823900    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="config-reloader"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823920    5600 cpu_manager.go:414] "containerMap: removing container" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="init-config-reloader"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823926    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="init-config-reloader"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823937    5600 cpu_manager.go:414] "containerMap: removing container" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy-web"
Nov 03 19:40:40.823943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.823944    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy-web"
Nov 03 19:40:40.824476 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.824004    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy-web"
Nov 03 19:40:40.824476 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.824016    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy-metric"
Nov 03 19:40:40.824476 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.824046    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="kube-rbac-proxy"
Nov 03 19:40:40.824476 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.824055    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="alertmanager"
Nov 03 19:40:40.824476 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.824063    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="prom-label-proxy"
Nov 03 19:40:40.824476 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.824072    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="config-reloader"
Nov 03 19:40:40.824476 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.824079    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" containerName="init-config-reloader"
Nov 03 19:40:40.830485 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.830468    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.835040 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.835019    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 19:40:40.835140 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.835053    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 19:40:40.835140 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.835115    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 19:40:40.835236 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.835186    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 19:40:40.835236 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.835206    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 19:40:40.835236 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.835210    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 19:40:40.835337 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.835220    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 19:40:40.835337 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.835293    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kcnh9\""
Nov 03 19:40:40.835398 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.835348    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 19:40:40.839810 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.839794    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 19:40:40.844298 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.844268    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:40:40.994542 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.994424    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4a52ed9-95d8-4b79-b0f2-8c230139391b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.994542 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.994496    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4a52ed9-95d8-4b79-b0f2-8c230139391b-config-out\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.995026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.994587    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4a52ed9-95d8-4b79-b0f2-8c230139391b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.995026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.994622    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.995026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.994693    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.995026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.994729    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.995026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.994748    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a52ed9-95d8-4b79-b0f2-8c230139391b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.995026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.994840    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4a52ed9-95d8-4b79-b0f2-8c230139391b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.995026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.994877    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.995026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.994909    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.995026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.994942    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.995026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.995004    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplwv\" (UniqueName: \"kubernetes.io/projected/b4a52ed9-95d8-4b79-b0f2-8c230139391b-kube-api-access-dplwv\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:40.995337 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:40.995034    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-web-config\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096047 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096003    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096204 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096055    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096204 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096081    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096204 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096107    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a52ed9-95d8-4b79-b0f2-8c230139391b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096204 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096144    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4a52ed9-95d8-4b79-b0f2-8c230139391b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096204 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096171    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096226    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096271    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096313    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplwv\" (UniqueName: \"kubernetes.io/projected/b4a52ed9-95d8-4b79-b0f2-8c230139391b-kube-api-access-dplwv\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096346    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-web-config\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096374    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4a52ed9-95d8-4b79-b0f2-8c230139391b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096411    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4a52ed9-95d8-4b79-b0f2-8c230139391b-config-out\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096474    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4a52ed9-95d8-4b79-b0f2-8c230139391b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.096850 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096519    5600 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:40:41.096850 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.096802    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4a52ed9-95d8-4b79-b0f2-8c230139391b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.097375 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.097310    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4a52ed9-95d8-4b79-b0f2-8c230139391b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.097702 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.097675    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a52ed9-95d8-4b79-b0f2-8c230139391b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.099527 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.099482    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.099841 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.099822    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.099974 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.099945    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.100086 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.099977    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-web-config\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.100086 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.100059    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4a52ed9-95d8-4b79-b0f2-8c230139391b-config-out\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.100324 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.100302    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.100464 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.100380    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.101069 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.101051    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4a52ed9-95d8-4b79-b0f2-8c230139391b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.101514 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.101495    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4a52ed9-95d8-4b79-b0f2-8c230139391b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.104737 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.104716    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplwv\" (UniqueName: \"kubernetes.io/projected/b4a52ed9-95d8-4b79-b0f2-8c230139391b-kube-api-access-dplwv\") pod \"alertmanager-main-0\" (UID: \"b4a52ed9-95d8-4b79-b0f2-8c230139391b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.158647 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.158609    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:40:41.438620 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.438588    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:40:41.440384 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:41.440359    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a52ed9_95d8_4b79_b0f2_8c230139391b.slice/crio-fd23e87d6d5320d985a49258fb2c384155602996d0315f29f50924543b1ebd14 WatchSource:0}: Error finding container fd23e87d6d5320d985a49258fb2c384155602996d0315f29f50924543b1ebd14: Status 404 returned error can't find the container with id fd23e87d6d5320d985a49258fb2c384155602996d0315f29f50924543b1ebd14
Nov 03 19:40:41.614548 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.614515    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd1ca4f-1929-48df-8769-b2e469863d7a" path="/var/lib/kubelet/pods/1fd1ca4f-1929-48df-8769-b2e469863d7a/volumes"
Nov 03 19:40:41.773825 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.773740    5600 generic.go:356] "Generic (PLEG): container finished" podID="b4a52ed9-95d8-4b79-b0f2-8c230139391b" containerID="8c007724af33501d82a7fd590efd9c08922e1c0935cddd358b7272dd816dfddb" exitCode=0
Nov 03 19:40:41.773825 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.773789    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4a52ed9-95d8-4b79-b0f2-8c230139391b","Type":"ContainerDied","Data":"8c007724af33501d82a7fd590efd9c08922e1c0935cddd358b7272dd816dfddb"}
Nov 03 19:40:41.774007 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:41.773844    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4a52ed9-95d8-4b79-b0f2-8c230139391b","Type":"ContainerStarted","Data":"fd23e87d6d5320d985a49258fb2c384155602996d0315f29f50924543b1ebd14"}
Nov 03 19:40:42.780113 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.780073    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4a52ed9-95d8-4b79-b0f2-8c230139391b","Type":"ContainerStarted","Data":"a10494f24f817a9ac58bd67e5b2b7a89d8ececa4efb7d4c9e65a4d22a0bc0dea"}
Nov 03 19:40:42.780529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.780164    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4a52ed9-95d8-4b79-b0f2-8c230139391b","Type":"ContainerStarted","Data":"fe1b77ef7b7ea1d38d6db3b357f221bb3467244b84e67b159536906f72363c68"}
Nov 03 19:40:42.780529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.780194    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4a52ed9-95d8-4b79-b0f2-8c230139391b","Type":"ContainerStarted","Data":"fc661b862f46a62ac86589e82ba5c3eccfcc7613c500efa6e7b8da18888e790b"}
Nov 03 19:40:42.780529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.780222    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4a52ed9-95d8-4b79-b0f2-8c230139391b","Type":"ContainerStarted","Data":"14c096b738a964f0275e34391c5c92f1501d50c5497c3daf07097a8b46941596"}
Nov 03 19:40:42.780529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.780246    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4a52ed9-95d8-4b79-b0f2-8c230139391b","Type":"ContainerStarted","Data":"6a836a4d5acd1815767543ebf9566dcedef4b01dcc2881d73af90cdf1a8bce7a"}
Nov 03 19:40:42.780529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.780268    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4a52ed9-95d8-4b79-b0f2-8c230139391b","Type":"ContainerStarted","Data":"0138df24fb90ece7cd4c30cf96d220c10ed5398b33366b15e7020b854e0c4c7c"}
Nov 03 19:40:42.807034 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.806920    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.793894818 podStartE2EDuration="2.806906536s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.011718ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:42.805462464 +0000 UTC m=+25.318284247" watchObservedRunningTime="2026-11-03 19:40:42.806906536 +0000 UTC m=+25.319728312"
Nov 03 19:40:42.940827 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.940783    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:40:42.941169 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.941140    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="prometheus" containerID="cri-o://5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2" gracePeriod=600
Nov 03 19:40:42.941245 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.941188    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy-thanos" containerID="cri-o://15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686" gracePeriod=600
Nov 03 19:40:42.941245 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.941181    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy" containerID="cri-o://37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1" gracePeriod=600
Nov 03 19:40:42.941245 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.941221    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="thanos-sidecar" containerID="cri-o://6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a" gracePeriod=600
Nov 03 19:40:42.941386 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.941248    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="config-reloader" containerID="cri-o://b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118" gracePeriod=600
Nov 03 19:40:42.941386 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:42.941269    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy-web" containerID="cri-o://c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6" gracePeriod=600
Nov 03 19:40:43.787818 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:43.787776    5600 generic.go:356] "Generic (PLEG): container finished" podID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerID="15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686" exitCode=0
Nov 03 19:40:43.787818 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:43.787805    5600 generic.go:356] "Generic (PLEG): container finished" podID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerID="37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1" exitCode=0
Nov 03 19:40:43.787818 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:43.787814    5600 generic.go:356] "Generic (PLEG): container finished" podID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerID="6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a" exitCode=0
Nov 03 19:40:43.787818 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:43.787822    5600 generic.go:356] "Generic (PLEG): container finished" podID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerID="b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118" exitCode=0
Nov 03 19:40:43.787818 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:43.787829    5600 generic.go:356] "Generic (PLEG): container finished" podID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerID="5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2" exitCode=0
Nov 03 19:40:43.788501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:43.787845    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerDied","Data":"15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686"}
Nov 03 19:40:43.788501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:43.787907    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerDied","Data":"37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1"}
Nov 03 19:40:43.788501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:43.787933    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerDied","Data":"6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a"}
Nov 03 19:40:43.788501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:43.787959    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerDied","Data":"b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118"}
Nov 03 19:40:43.788501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:43.787983    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerDied","Data":"5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2"}
Nov 03 19:40:44.183822 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.183799    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.221745 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.221711    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-tls\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.221883 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.221767    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.221883 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.221801    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-kube-rbac-proxy\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.221883 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.221835    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-metrics-client-ca\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.221883 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.221867    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-2\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222117 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.221925    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-web-config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-web-config\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222117 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.221960    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222117 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.221990    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-config\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222117 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222017    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-thanos-prometheus-http-client-file\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222117 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222055    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-db\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222117 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222096    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-kube-api-access-tqrdl\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-kube-api-access-tqrdl\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222376 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222132    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-config-out\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222376 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222171    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-metrics-client-certs\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222376 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222216    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-serving-certs-ca-bundle\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222376 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222252    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-grpc-tls\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222376 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222294    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-tls-assets\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-tls-assets\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222376 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222324    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-1\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222376 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222371    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-trusted-ca-bundle\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222920 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222409    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-kubelet-serving-ca-bundle\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.222920 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.222465    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-0\") pod \"9d7d54f2-9efe-467f-82a4-693e30e17b58\" (UID: \"9d7d54f2-9efe-467f-82a4-693e30e17b58\") "
Nov 03 19:40:44.223101 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.223066    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-2" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:44.223856 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.223518    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-metrics-client-ca" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:44.225165 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.224882    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-kube-rbac-proxy" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:44.225165 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.224920    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-1" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:44.225816 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.225625    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-db" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:40:44.225816 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.225743    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-0" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:44.226511 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.226485    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-trusted-ca-bundle" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:44.227525 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.227454    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-tls" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:44.227877 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.227850    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-kubelet-serving-ca-bundle" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:44.228007 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.227959    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-serving-certs-ca-bundle" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:44.229839 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.229802    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-kube-rbac-proxy-web" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:44.230489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.230452    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-thanos-sidecar-tls" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:44.231139 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.231116    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-config-out" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:40:44.232385 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.232360    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-metrics-client-certs" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:44.232820 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.232777    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-grpc-tls" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:44.232997 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.232962    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-tls-assets" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:40:44.233080 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.233049    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-config" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:44.233080 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.233054    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-thanos-prometheus-http-client-file" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:44.234220 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.234188    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-kube-api-access-tqrdl" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "kube-api-access-tqrdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:40:44.239406 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.239381    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-web-config" pod "9d7d54f2-9efe-467f-82a4-693e30e17b58" (UID: "9d7d54f2-9efe-467f-82a4-693e30e17b58"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:44.323384 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323317    5600 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323384 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323342    5600 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-grpc-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323384 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323352    5600 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-tls-assets\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323384 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323363    5600 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323384 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323373    5600 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323384 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323381    5600 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323389    5600 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323398    5600 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323407    5600 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323416    5600 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-kube-rbac-proxy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323424    5600 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-configmap-metrics-client-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323452    5600 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323462    5600 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-web-config\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323472    5600 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323481    5600 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-config\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323489    5600 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323497    5600 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-prometheus-k8s-db\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323505    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqrdl\" (UniqueName: \"kubernetes.io/projected/9d7d54f2-9efe-467f-82a4-693e30e17b58-kube-api-access-tqrdl\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323513    5600 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d7d54f2-9efe-467f-82a4-693e30e17b58-config-out\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.323662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.323521    5600 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d7d54f2-9efe-467f-82a4-693e30e17b58-secret-metrics-client-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:44.793909 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.793871    5600 generic.go:356] "Generic (PLEG): container finished" podID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerID="c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6" exitCode=0
Nov 03 19:40:44.794309 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.793954    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerDied","Data":"c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6"}
Nov 03 19:40:44.794309 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.793960    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.794309 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.794049    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d7d54f2-9efe-467f-82a4-693e30e17b58","Type":"ContainerDied","Data":"cb232ad37c2c4153751d97b8e4c603375bdb7d55824e2f98d0005baec81a3471"}
Nov 03 19:40:44.794309 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.794083    5600 scope.go:122] "RemoveContainer" containerID="15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686"
Nov 03 19:40:44.802448 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.802413    5600 scope.go:122] "RemoveContainer" containerID="37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1"
Nov 03 19:40:44.808942 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.808929    5600 scope.go:122] "RemoveContainer" containerID="c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6"
Nov 03 19:40:44.815216 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.815202    5600 scope.go:122] "RemoveContainer" containerID="6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a"
Nov 03 19:40:44.818792 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.818771    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:40:44.822499 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.822478    5600 scope.go:122] "RemoveContainer" containerID="b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118"
Nov 03 19:40:44.823419 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.823369    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:40:44.829206 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.829187    5600 scope.go:122] "RemoveContainer" containerID="5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2"
Nov 03 19:40:44.836062 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.836045    5600 scope.go:122] "RemoveContainer" containerID="e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f"
Nov 03 19:40:44.842855 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.842837    5600 scope.go:122] "RemoveContainer" containerID="15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686"
Nov 03 19:40:44.843109 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:44.843093    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686\": container with ID starting with 15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686 not found: ID does not exist" containerID="15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686"
Nov 03 19:40:44.843155 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.843124    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686"} err="failed to get container status \"15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686\": rpc error: code = NotFound desc = could not find container \"15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686\": container with ID starting with 15373d4f4ff00f36c239c947119c6854169045452457b5a9ba7e9722747e3686 not found: ID does not exist"
Nov 03 19:40:44.843155 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.843142    5600 scope.go:122] "RemoveContainer" containerID="37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1"
Nov 03 19:40:44.843366 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:44.843343    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1\": container with ID starting with 37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1 not found: ID does not exist" containerID="37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1"
Nov 03 19:40:44.843412 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.843374    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1"} err="failed to get container status \"37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1\": rpc error: code = NotFound desc = could not find container \"37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1\": container with ID starting with 37bbf370a85ae7f57dfa24da039697607b9fd53fc9b7c102ed68d688daadb1a1 not found: ID does not exist"
Nov 03 19:40:44.843412 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.843389    5600 scope.go:122] "RemoveContainer" containerID="c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6"
Nov 03 19:40:44.843608 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:44.843594    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6\": container with ID starting with c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6 not found: ID does not exist" containerID="c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6"
Nov 03 19:40:44.843660 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.843611    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6"} err="failed to get container status \"c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6\": rpc error: code = NotFound desc = could not find container \"c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6\": container with ID starting with c629fe9f4237a5be681ca42ee4ea123f1587404f9d8d03cde8617f8b4c1ff5f6 not found: ID does not exist"
Nov 03 19:40:44.843660 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.843623    5600 scope.go:122] "RemoveContainer" containerID="6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a"
Nov 03 19:40:44.843838 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:44.843819    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a\": container with ID starting with 6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a not found: ID does not exist" containerID="6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a"
Nov 03 19:40:44.843880 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.843841    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a"} err="failed to get container status \"6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a\": rpc error: code = NotFound desc = could not find container \"6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a\": container with ID starting with 6e7db1bab5ff7f77992aa601154c3dcd5e9a6c9428994fec6ba8b4991ca49b2a not found: ID does not exist"
Nov 03 19:40:44.843880 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.843854    5600 scope.go:122] "RemoveContainer" containerID="b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118"
Nov 03 19:40:44.844106 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:44.844087    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118\": container with ID starting with b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118 not found: ID does not exist" containerID="b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118"
Nov 03 19:40:44.844184 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.844114    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118"} err="failed to get container status \"b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118\": rpc error: code = NotFound desc = could not find container \"b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118\": container with ID starting with b32c4ce546b16ce4785b5ae878b795fa44667e7ae22a7e51484d9e67d6ae0118 not found: ID does not exist"
Nov 03 19:40:44.844184 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.844138    5600 scope.go:122] "RemoveContainer" containerID="5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2"
Nov 03 19:40:44.844388 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:44.844369    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2\": container with ID starting with 5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2 not found: ID does not exist" containerID="5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2"
Nov 03 19:40:44.844454 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.844392    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2"} err="failed to get container status \"5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2\": rpc error: code = NotFound desc = could not find container \"5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2\": container with ID starting with 5df193179cc2d79df939912b1acb838b28fec20c153064e6a5719827292285f2 not found: ID does not exist"
Nov 03 19:40:44.844454 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.844405    5600 scope.go:122] "RemoveContainer" containerID="e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f"
Nov 03 19:40:44.844697 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:44.844676    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f\": container with ID starting with e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f not found: ID does not exist" containerID="e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f"
Nov 03 19:40:44.844737 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.844703    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f"} err="failed to get container status \"e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f\": rpc error: code = NotFound desc = could not find container \"e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f\": container with ID starting with e38517025b9fe634f21166037be0ee74b7bdda0a23820ae6f1c67eae7dd5bb6f not found: ID does not exist"
Nov 03 19:40:44.850082 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850060    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:40:44.850337 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850324    5600 cpu_manager.go:414] "containerMap: removing container" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="thanos-sidecar"
Nov 03 19:40:44.850383 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850339    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="thanos-sidecar"
Nov 03 19:40:44.850383 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850351    5600 cpu_manager.go:414] "containerMap: removing container" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy-thanos"
Nov 03 19:40:44.850383 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850357    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy-thanos"
Nov 03 19:40:44.850383 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850374    5600 cpu_manager.go:414] "containerMap: removing container" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy"
Nov 03 19:40:44.850383 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850380    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850390    5600 cpu_manager.go:414] "containerMap: removing container" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="prometheus"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850396    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="prometheus"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850455    5600 cpu_manager.go:414] "containerMap: removing container" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="init-config-reloader"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850460    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="init-config-reloader"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850469    5600 cpu_manager.go:414] "containerMap: removing container" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy-web"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850474    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy-web"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850481    5600 cpu_manager.go:414] "containerMap: removing container" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="config-reloader"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850487    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="config-reloader"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850531    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850541    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="prometheus"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850546    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="init-config-reloader"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850552    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy-web"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850557    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="config-reloader"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850561    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="thanos-sidecar"
Nov 03 19:40:44.850563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.850568    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" containerName="kube-rbac-proxy-thanos"
Nov 03 19:40:44.853669 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.853654    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.857640 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.857619    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 19:40:44.857640 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.857631    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 19:40:44.857640 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.857640    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 19:40:44.860819 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.860802    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 19:40:44.860908 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.860892    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 19:40:44.862213 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.862197    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 19:40:44.862446 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.862408    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 19:40:44.862446 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.862421    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 19:40:44.862577 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.862424    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 19:40:44.862577 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.862428    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 19:40:44.862577 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.862464    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d5pj80fngeohf\""
Nov 03 19:40:44.863599 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.863580    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 19:40:44.863778 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.863764    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x7mbg\""
Nov 03 19:40:44.864951 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.864937    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 19:40:44.865124 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.865114    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 19:40:44.865325 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.865312    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 19:40:44.869193 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.869174    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:40:44.928590 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928554    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.928590 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928591    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df71796f-bae7-44f2-b35a-dad7a3f03497-config-out\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.928787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928609    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5hl\" (UniqueName: \"kubernetes.io/projected/df71796f-bae7-44f2-b35a-dad7a3f03497-kube-api-access-bt5hl\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.928787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928672    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.928787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928702    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.928787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928737    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.928928 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928820    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.928928 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928847    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.928928 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928871    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.928928 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928888    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.928928 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928913    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-web-config\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.929092 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.928954    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.929092 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.929013    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.929092 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.929034    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.929092 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.929050    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.929092 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.929064    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.929092 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.929086    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df71796f-bae7-44f2-b35a-dad7a3f03497-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.929283 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.929142    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.929283 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.929187    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-config\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:44.929283 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:44.929222    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030012 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.029907    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030040    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030068    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030114 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030084    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030284 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030112    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-web-config\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030284 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030141    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030284 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030192    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030284 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030226    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030284 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030247    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030284 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030269    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030611 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030303    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df71796f-bae7-44f2-b35a-dad7a3f03497-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030611 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030330    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030611 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030356    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-config\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030611 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030383    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030611 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030425    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030611 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030498    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df71796f-bae7-44f2-b35a-dad7a3f03497-config-out\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030611 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030522    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5hl\" (UniqueName: \"kubernetes.io/projected/df71796f-bae7-44f2-b35a-dad7a3f03497-kube-api-access-bt5hl\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030611 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030553    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030611 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030578    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.030611 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030609    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.031101 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.030756    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.031170 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.031142    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.031394 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.031366    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.032116 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.032090    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.032782 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.032469    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.032782 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.032737    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.033028 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.032999    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.033629 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.033582    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.033718 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.033673    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.033799 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.033762    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-web-config\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.035480 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.035368    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/df71796f-bae7-44f2-b35a-dad7a3f03497-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.035725 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.035707    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df71796f-bae7-44f2-b35a-dad7a3f03497-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.035910 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.035889    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.035968 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.035946    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-config\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.036452 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.036411    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.036452 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.036417    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.036562 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.036469    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.036707 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.036690    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df71796f-bae7-44f2-b35a-dad7a3f03497-config-out\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.037128 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.037092    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df71796f-bae7-44f2-b35a-dad7a3f03497-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.039145 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.039129    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5hl\" (UniqueName: \"kubernetes.io/projected/df71796f-bae7-44f2-b35a-dad7a3f03497-kube-api-access-bt5hl\") pod \"prometheus-k8s-0\" (UID: \"df71796f-bae7-44f2-b35a-dad7a3f03497\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.179675 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.179641    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:45.460553 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.460411    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:40:45.462534 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:40:45.462502    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf71796f_bae7_44f2_b35a_dad7a3f03497.slice/crio-8d9aecb1335b7a361ce9c5ef39b646aa396b9768d17f252c2cd46bc1e74ec6cf WatchSource:0}: Error finding container 8d9aecb1335b7a361ce9c5ef39b646aa396b9768d17f252c2cd46bc1e74ec6cf: Status 404 returned error can't find the container with id 8d9aecb1335b7a361ce9c5ef39b646aa396b9768d17f252c2cd46bc1e74ec6cf
Nov 03 19:40:45.613305 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.613273    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7d54f2-9efe-467f-82a4-693e30e17b58" path="/var/lib/kubelet/pods/9d7d54f2-9efe-467f-82a4-693e30e17b58/volumes"
Nov 03 19:40:45.800069 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.799991    5600 generic.go:356] "Generic (PLEG): container finished" podID="df71796f-bae7-44f2-b35a-dad7a3f03497" containerID="d81fddde463a79422afb75800604b0542e6986938c9c5ee8e86b27c4cec5bfe6" exitCode=0
Nov 03 19:40:45.800069 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.800043    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df71796f-bae7-44f2-b35a-dad7a3f03497","Type":"ContainerDied","Data":"d81fddde463a79422afb75800604b0542e6986938c9c5ee8e86b27c4cec5bfe6"}
Nov 03 19:40:45.800462 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:45.800118    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df71796f-bae7-44f2-b35a-dad7a3f03497","Type":"ContainerStarted","Data":"8d9aecb1335b7a361ce9c5ef39b646aa396b9768d17f252c2cd46bc1e74ec6cf"}
Nov 03 19:40:46.806947 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:46.806908    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df71796f-bae7-44f2-b35a-dad7a3f03497","Type":"ContainerStarted","Data":"11d49c5675942ed0ffc00b1afa856316c50b204aab1f74c47012a114c68f6217"}
Nov 03 19:40:46.807394 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:46.806976    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df71796f-bae7-44f2-b35a-dad7a3f03497","Type":"ContainerStarted","Data":"cebb87d0f424040b0ce4bc3d5dc782bd763040713f84d3872bdc2e9a13d457ff"}
Nov 03 19:40:46.807394 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:46.807003    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df71796f-bae7-44f2-b35a-dad7a3f03497","Type":"ContainerStarted","Data":"9dcabefd4b80d339326718fe61c0f7b8462a409b6589c099379a265925aa32b9"}
Nov 03 19:40:46.807394 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:46.807025    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df71796f-bae7-44f2-b35a-dad7a3f03497","Type":"ContainerStarted","Data":"1c599a27a432e9ed9f71e7c677dd2ab0d6034db9336e54f00de115b0af675f18"}
Nov 03 19:40:46.807394 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:46.807046    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df71796f-bae7-44f2-b35a-dad7a3f03497","Type":"ContainerStarted","Data":"c6a3f5a1b0734ff0b5de1c68b90f34a83231a11dca384c4a9e6ac7e7e0bc2ae2"}
Nov 03 19:40:46.807394 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:46.807070    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df71796f-bae7-44f2-b35a-dad7a3f03497","Type":"ContainerStarted","Data":"3efff8967f854b73e2a63645cad2ec126e1b3d89be1c970ed1e44a8d18b08978"}
Nov 03 19:40:46.835110 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:46.835056    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.820691274 podStartE2EDuration="2.835043098s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.351824ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:46.833633538 +0000 UTC m=+29.346455319" watchObservedRunningTime="2026-11-03 19:40:46.835043098 +0000 UTC m=+29.347864877"
Nov 03 19:40:50.180636 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:50.180597    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:52.236994 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:52.236957    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:52.253033 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:52.253003    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:52.843556 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:52.843530    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:40:53.206182 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:40:53.206153    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9bwkj"
Nov 03 19:40:58.794799 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:40:58.794760    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:41:28.793713 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:41:28.793676    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:41:58.793949 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:41:58.793911    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:42:28.792649 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:42:28.792611    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:42:58.794168 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:42:58.794126    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:43:28.794013 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:43:28.793935    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:43:58.794230 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:43:58.794191    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:44:28.793842 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:44:28.793801    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:44:58.793468 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:44:58.793368    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:45:10.748925 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.748871    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"]
Nov 03 19:45:10.760056 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.760013    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"]
Nov 03 19:45:10.760228 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.760144    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:10.765271 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.765250    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 19:45:10.765477 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.765286    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-7pq7d\""
Nov 03 19:45:10.765613 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.765332    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 19:45:10.810529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.810495    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:10.810529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.810534    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqk4z\" (UniqueName: \"kubernetes.io/projected/d74a273d-d3af-4738-97d0-f3bb2f193b5f-kube-api-access-zqk4z\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:10.810725 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.810552    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:10.911926 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.911891    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:10.912118 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.911942    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqk4z\" (UniqueName: \"kubernetes.io/projected/d74a273d-d3af-4738-97d0-f3bb2f193b5f-kube-api-access-zqk4z\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:10.912118 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.911974    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:10.912297 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.912277    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:10.912391 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.912370    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:10.920618 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:10.920588    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqk4z\" (UniqueName: \"kubernetes.io/projected/d74a273d-d3af-4738-97d0-f3bb2f193b5f-kube-api-access-zqk4z\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:11.080324 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:11.080239    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:11.350945 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:11.350799    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"]
Nov 03 19:45:11.353527 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:45:11.353500    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74a273d_d3af_4738_97d0_f3bb2f193b5f.slice/crio-fc301d9850898324493bda471ff20192e7cbe251d159cca38099cb6eb146b124 WatchSource:0}: Error finding container fc301d9850898324493bda471ff20192e7cbe251d159cca38099cb6eb146b124: Status 404 returned error can't find the container with id fc301d9850898324493bda471ff20192e7cbe251d159cca38099cb6eb146b124
Nov 03 19:45:11.355520 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:11.355499    5600 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:45:11.698303 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:11.698273    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc" event={"ID":"d74a273d-d3af-4738-97d0-f3bb2f193b5f","Type":"ContainerStarted","Data":"fc301d9850898324493bda471ff20192e7cbe251d159cca38099cb6eb146b124"}
Nov 03 19:45:12.840703 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:12.840672    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"]
Nov 03 19:45:12.844172 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:12.844135    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:12.849617 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:12.849594    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-nwvj9\""
Nov 03 19:45:12.854051 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:12.853989    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"]
Nov 03 19:45:12.929824 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:12.929791    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:12.930009 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:12.929849    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wql\" (UniqueName: \"kubernetes.io/projected/400d53f3-892f-4726-bc5c-8ecf1c503846-kube-api-access-f2wql\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:12.930009 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:12.929962    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:12.944309 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:12.944275    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"]
Nov 03 19:45:12.947862 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:12.947843    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:12.951124 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:12.951066    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-qf4wd\""
Nov 03 19:45:12.955770 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:12.955748    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"]
Nov 03 19:45:13.030823 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.030784    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:13.031004 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.030850    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wql\" (UniqueName: \"kubernetes.io/projected/400d53f3-892f-4726-bc5c-8ecf1c503846-kube-api-access-f2wql\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:13.031004 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.030882    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:13.031004 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.030901    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:13.031004 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.030958    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:13.031004 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.030982    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q9f7\" (UniqueName: \"kubernetes.io/projected/db404f1b-c1b6-4b18-a5f6-16feb0de038a-kube-api-access-8q9f7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:13.031448 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.031397    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:13.031542 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.031529    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:13.041454 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.041411    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wql\" (UniqueName: \"kubernetes.io/projected/400d53f3-892f-4726-bc5c-8ecf1c503846-kube-api-access-f2wql\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:13.041588 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.041561    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"]
Nov 03 19:45:13.045201 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.045177    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:13.051566 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.051545    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-jmvc5\""
Nov 03 19:45:13.053072 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.053048    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"]
Nov 03 19:45:13.131623 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.131544    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:13.131623 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.131584    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q9f7\" (UniqueName: \"kubernetes.io/projected/db404f1b-c1b6-4b18-a5f6-16feb0de038a-kube-api-access-8q9f7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:13.131623 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.131606    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:13.131832 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.131661    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:13.131832 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.131683    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:13.131832 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.131727    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2vf4\" (UniqueName: \"kubernetes.io/projected/568b5e19-06b2-482d-877f-66704c6d1b0b-kube-api-access-w2vf4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:13.132074 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.132058    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:13.132563 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.132537    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:13.138680 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.138656    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"]
Nov 03 19:45:13.142173 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.142152    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:13.144566 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.144541    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q9f7\" (UniqueName: \"kubernetes.io/projected/db404f1b-c1b6-4b18-a5f6-16feb0de038a-kube-api-access-8q9f7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:13.145910 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.145890    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-45wmw\""
Nov 03 19:45:13.150454 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.150362    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"]
Nov 03 19:45:13.163797 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.163771    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:13.234916 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.234883    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2vf4\" (UniqueName: \"kubernetes.io/projected/568b5e19-06b2-482d-877f-66704c6d1b0b-kube-api-access-w2vf4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:13.235061 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.234964    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:13.235061 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.235040    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:13.235179 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.235069    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:13.235179 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.235103    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:13.235179 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.235139    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbxg\" (UniqueName: \"kubernetes.io/projected/36298071-9373-4c44-b50f-32d783144cbf-kube-api-access-wnbxg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:13.236146 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.235782    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:13.236146 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.235997    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:13.244959 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.244493    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2vf4\" (UniqueName: \"kubernetes.io/projected/568b5e19-06b2-482d-877f-66704c6d1b0b-kube-api-access-w2vf4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:13.266143 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.266118    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:13.336174 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.336119    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:13.336499 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.336187    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:13.336499 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.336228    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbxg\" (UniqueName: \"kubernetes.io/projected/36298071-9373-4c44-b50f-32d783144cbf-kube-api-access-wnbxg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:13.336638 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.336593    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:13.336827 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.336808    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:13.346427 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.346400    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbxg\" (UniqueName: \"kubernetes.io/projected/36298071-9373-4c44-b50f-32d783144cbf-kube-api-access-wnbxg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:13.366091 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.366063    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:13.456424 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.456393    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"]
Nov 03 19:45:13.460000 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:45:13.459964    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400d53f3_892f_4726_bc5c_8ecf1c503846.slice/crio-87e8c9207173bf508f1e7b52397642b782600f86b52d548f96ef3acdc1431490 WatchSource:0}: Error finding container 87e8c9207173bf508f1e7b52397642b782600f86b52d548f96ef3acdc1431490: Status 404 returned error can't find the container with id 87e8c9207173bf508f1e7b52397642b782600f86b52d548f96ef3acdc1431490
Nov 03 19:45:13.466098 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.466077    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:13.555833 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.555803    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"]
Nov 03 19:45:13.562957 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:45:13.562926    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb404f1b_c1b6_4b18_a5f6_16feb0de038a.slice/crio-c2fadf8bef49aff62d30dd9d1efe4ae47db50eb7587d897d2ff21ef33013a57e WatchSource:0}: Error finding container c2fadf8bef49aff62d30dd9d1efe4ae47db50eb7587d897d2ff21ef33013a57e: Status 404 returned error can't find the container with id c2fadf8bef49aff62d30dd9d1efe4ae47db50eb7587d897d2ff21ef33013a57e
Nov 03 19:45:13.645823 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.645795    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"]
Nov 03 19:45:13.649710 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:45:13.649685    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568b5e19_06b2_482d_877f_66704c6d1b0b.slice/crio-819fd902f8569722fad6e76973c373a04c749172944350a1a862e7e183c945db WatchSource:0}: Error finding container 819fd902f8569722fad6e76973c373a04c749172944350a1a862e7e183c945db: Status 404 returned error can't find the container with id 819fd902f8569722fad6e76973c373a04c749172944350a1a862e7e183c945db
Nov 03 19:45:13.707142 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.707054    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d" event={"ID":"400d53f3-892f-4726-bc5c-8ecf1c503846","Type":"ContainerStarted","Data":"87e8c9207173bf508f1e7b52397642b782600f86b52d548f96ef3acdc1431490"}
Nov 03 19:45:13.708402 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.708374    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn" event={"ID":"568b5e19-06b2-482d-877f-66704c6d1b0b","Type":"ContainerStarted","Data":"819fd902f8569722fad6e76973c373a04c749172944350a1a862e7e183c945db"}
Nov 03 19:45:13.709787 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.709760    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb" event={"ID":"db404f1b-c1b6-4b18-a5f6-16feb0de038a","Type":"ContainerStarted","Data":"c2fadf8bef49aff62d30dd9d1efe4ae47db50eb7587d897d2ff21ef33013a57e"}
Nov 03 19:45:13.750824 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:13.750803    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"]
Nov 03 19:45:13.752248 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:45:13.752223    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36298071_9373_4c44_b50f_32d783144cbf.slice/crio-aa6b4d02246d95b2d4af4e07bdf6fa3b4557569790ba013c4d100d66d36e7be7 WatchSource:0}: Error finding container aa6b4d02246d95b2d4af4e07bdf6fa3b4557569790ba013c4d100d66d36e7be7: Status 404 returned error can't find the container with id aa6b4d02246d95b2d4af4e07bdf6fa3b4557569790ba013c4d100d66d36e7be7
Nov 03 19:45:14.555891 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.555859    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"]
Nov 03 19:45:14.560470 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.560425    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:14.568016 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.567956    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"]
Nov 03 19:45:14.568577 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.568545    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-hv5kv\""
Nov 03 19:45:14.649348 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.649308    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:14.649571 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.649373    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mst66\" (UniqueName: \"kubernetes.io/projected/40b6a1b6-a798-46af-b02c-49314d794a52-kube-api-access-mst66\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:14.649571 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.649544    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:14.723119 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.723080    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv" event={"ID":"36298071-9373-4c44-b50f-32d783144cbf","Type":"ContainerStarted","Data":"aa6b4d02246d95b2d4af4e07bdf6fa3b4557569790ba013c4d100d66d36e7be7"}
Nov 03 19:45:14.750176 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.750138    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:14.750356 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.750236    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:14.750356 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.750283    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mst66\" (UniqueName: \"kubernetes.io/projected/40b6a1b6-a798-46af-b02c-49314d794a52-kube-api-access-mst66\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:14.750913 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.750892    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:14.751077 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.751063    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:14.765003 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.764953    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mst66\" (UniqueName: \"kubernetes.io/projected/40b6a1b6-a798-46af-b02c-49314d794a52-kube-api-access-mst66\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:14.885568 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:14.885267    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:15.184065 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:15.183912    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"]
Nov 03 19:45:15.734311 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:15.734271    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj" event={"ID":"40b6a1b6-a798-46af-b02c-49314d794a52","Type":"ContainerStarted","Data":"55bba11c47a62efe1e3c5cddb7e761f7a9aaa6c8ca9a3d2dea1728f3c4430663"}
Nov 03 19:45:16.068424 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.068296    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"]
Nov 03 19:45:16.073982 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.073954    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:16.092097 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.092078    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-wtc8j\""
Nov 03 19:45:16.092651 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.092627    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"]
Nov 03 19:45:16.164405 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.164360    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsns5\" (UniqueName: \"kubernetes.io/projected/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-kube-api-access-fsns5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:16.164613 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.164513    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:16.164722 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.164685    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:16.265603 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.265563    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:16.265795 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.265657    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:16.265795 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.265698    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsns5\" (UniqueName: \"kubernetes.io/projected/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-kube-api-access-fsns5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:16.266701 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.266673    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:16.266944 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.266924    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:16.281208 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.281157    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsns5\" (UniqueName: \"kubernetes.io/projected/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-kube-api-access-fsns5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:16.399919 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.399639    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:16.709967 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.709868    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"]
Nov 03 19:45:16.713737 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:45:16.713702    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a6b515_bcb9_4bc4_8012_dc4ef0894da3.slice/crio-ad24ac0ecaeffa21b055f9ff3360886b133082d4e36868387211adde2ff6779e WatchSource:0}: Error finding container ad24ac0ecaeffa21b055f9ff3360886b133082d4e36868387211adde2ff6779e: Status 404 returned error can't find the container with id ad24ac0ecaeffa21b055f9ff3360886b133082d4e36868387211adde2ff6779e
Nov 03 19:45:16.741587 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:16.741525    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp" event={"ID":"16a6b515-bcb9-4bc4-8012-dc4ef0894da3","Type":"ContainerStarted","Data":"ad24ac0ecaeffa21b055f9ff3360886b133082d4e36868387211adde2ff6779e"}
Nov 03 19:45:23.798302 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:23.798258    5600 generic.go:356] "Generic (PLEG): container finished" podID="d74a273d-d3af-4738-97d0-f3bb2f193b5f" containerID="2832354c2c4b914173ff7dacefbba57f4073ccc68144f37997f86ee75d1f543f" exitCode=0
Nov 03 19:45:23.798836 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:23.798337    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc" event={"ID":"d74a273d-d3af-4738-97d0-f3bb2f193b5f","Type":"ContainerDied","Data":"2832354c2c4b914173ff7dacefbba57f4073ccc68144f37997f86ee75d1f543f"}
Nov 03 19:45:24.807480 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:24.807405    5600 generic.go:356] "Generic (PLEG): container finished" podID="40b6a1b6-a798-46af-b02c-49314d794a52" containerID="bf648505a9baf9608429b092686ff728d461b5a164d196432bb784dadab9c3ed" exitCode=0
Nov 03 19:45:24.808135 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:24.807525    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj" event={"ID":"40b6a1b6-a798-46af-b02c-49314d794a52","Type":"ContainerDied","Data":"bf648505a9baf9608429b092686ff728d461b5a164d196432bb784dadab9c3ed"}
Nov 03 19:45:25.812994 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:25.812959    5600 generic.go:356] "Generic (PLEG): container finished" podID="400d53f3-892f-4726-bc5c-8ecf1c503846" containerID="69ad65bc15ce14422539a0a3dc54cc03d807179fa0a3d3284b38a563087ce2db" exitCode=0
Nov 03 19:45:25.813507 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:25.813055    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d" event={"ID":"400d53f3-892f-4726-bc5c-8ecf1c503846","Type":"ContainerDied","Data":"69ad65bc15ce14422539a0a3dc54cc03d807179fa0a3d3284b38a563087ce2db"}
Nov 03 19:45:25.814843 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:25.814814    5600 generic.go:356] "Generic (PLEG): container finished" podID="568b5e19-06b2-482d-877f-66704c6d1b0b" containerID="07c397426e20773f458ea425dc3ff2e2e359887c002eb7337116cd49d28ed4c0" exitCode=0
Nov 03 19:45:25.814968 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:25.814904    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn" event={"ID":"568b5e19-06b2-482d-877f-66704c6d1b0b","Type":"ContainerDied","Data":"07c397426e20773f458ea425dc3ff2e2e359887c002eb7337116cd49d28ed4c0"}
Nov 03 19:45:25.816762 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:25.816684    5600 generic.go:356] "Generic (PLEG): container finished" podID="16a6b515-bcb9-4bc4-8012-dc4ef0894da3" containerID="dcaced7b8c9a859a57198b72cc9ed618165030df48245fefe11db16ab4d8d120" exitCode=0
Nov 03 19:45:25.816762 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:25.816726    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp" event={"ID":"16a6b515-bcb9-4bc4-8012-dc4ef0894da3","Type":"ContainerDied","Data":"dcaced7b8c9a859a57198b72cc9ed618165030df48245fefe11db16ab4d8d120"}
Nov 03 19:45:26.824279 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:26.824239    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv" event={"ID":"36298071-9373-4c44-b50f-32d783144cbf","Type":"ContainerStarted","Data":"9e5d2e8b791161296b864079728cf49c770fd2d4c1191537f41dff3fe59b33d0"}
Nov 03 19:45:26.828333 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:26.827691    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp" event={"ID":"16a6b515-bcb9-4bc4-8012-dc4ef0894da3","Type":"ContainerStarted","Data":"638c2a56863e231ad7e6082929954bbd1898e0fe871e7e9ee4b944d659f3e711"}
Nov 03 19:45:26.830479 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:26.830113    5600 generic.go:356] "Generic (PLEG): container finished" podID="db404f1b-c1b6-4b18-a5f6-16feb0de038a" containerID="b1df30f3c0f41f71e8b59556e630abeba4f760933ce5fa2b12a7b2c3937ac986" exitCode=0
Nov 03 19:45:26.830479 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:26.830171    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb" event={"ID":"db404f1b-c1b6-4b18-a5f6-16feb0de038a","Type":"ContainerDied","Data":"b1df30f3c0f41f71e8b59556e630abeba4f760933ce5fa2b12a7b2c3937ac986"}
Nov 03 19:45:27.836262 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:27.836184    5600 generic.go:356] "Generic (PLEG): container finished" podID="400d53f3-892f-4726-bc5c-8ecf1c503846" containerID="956c4dfa61f4c6b496b47977b74875c55a72903516a20456b3005fbaf40e1b0c" exitCode=0
Nov 03 19:45:27.836726 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:27.836269    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d" event={"ID":"400d53f3-892f-4726-bc5c-8ecf1c503846","Type":"ContainerDied","Data":"956c4dfa61f4c6b496b47977b74875c55a72903516a20456b3005fbaf40e1b0c"}
Nov 03 19:45:27.839187 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:27.839162    5600 generic.go:356] "Generic (PLEG): container finished" podID="568b5e19-06b2-482d-877f-66704c6d1b0b" containerID="4997d1d5c68c5fd4e1942494107a686a22ca78c0ede8ba339940018f91849452" exitCode=0
Nov 03 19:45:27.839282 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:27.839254    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn" event={"ID":"568b5e19-06b2-482d-877f-66704c6d1b0b","Type":"ContainerDied","Data":"4997d1d5c68c5fd4e1942494107a686a22ca78c0ede8ba339940018f91849452"}
Nov 03 19:45:27.841847 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:27.841824    5600 generic.go:356] "Generic (PLEG): container finished" podID="40b6a1b6-a798-46af-b02c-49314d794a52" containerID="78794c0f5d21f918a95acc80d6c3ec41ac55cde7e2404c75433428bd0d0d7871" exitCode=0
Nov 03 19:45:27.841943 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:27.841879    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj" event={"ID":"40b6a1b6-a798-46af-b02c-49314d794a52","Type":"ContainerDied","Data":"78794c0f5d21f918a95acc80d6c3ec41ac55cde7e2404c75433428bd0d0d7871"}
Nov 03 19:45:27.844091 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:27.844073    5600 generic.go:356] "Generic (PLEG): container finished" podID="36298071-9373-4c44-b50f-32d783144cbf" containerID="9e5d2e8b791161296b864079728cf49c770fd2d4c1191537f41dff3fe59b33d0" exitCode=0
Nov 03 19:45:27.844197 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:27.844178    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv" event={"ID":"36298071-9373-4c44-b50f-32d783144cbf","Type":"ContainerDied","Data":"9e5d2e8b791161296b864079728cf49c770fd2d4c1191537f41dff3fe59b33d0"}
Nov 03 19:45:27.846380 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:27.846343    5600 generic.go:356] "Generic (PLEG): container finished" podID="16a6b515-bcb9-4bc4-8012-dc4ef0894da3" containerID="638c2a56863e231ad7e6082929954bbd1898e0fe871e7e9ee4b944d659f3e711" exitCode=0
Nov 03 19:45:27.846494 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:27.846382    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp" event={"ID":"16a6b515-bcb9-4bc4-8012-dc4ef0894da3","Type":"ContainerDied","Data":"638c2a56863e231ad7e6082929954bbd1898e0fe871e7e9ee4b944d659f3e711"}
Nov 03 19:45:28.808257 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:45:28.808157    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:45:29.862528 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:29.862488    5600 generic.go:356] "Generic (PLEG): container finished" podID="d74a273d-d3af-4738-97d0-f3bb2f193b5f" containerID="84046c2a16defdd8d2d4cb82a50dd77814bdf2856363029e7bfab1e0b1f92b9e" exitCode=0
Nov 03 19:45:29.863016 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:29.862987    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc" event={"ID":"d74a273d-d3af-4738-97d0-f3bb2f193b5f","Type":"ContainerDied","Data":"84046c2a16defdd8d2d4cb82a50dd77814bdf2856363029e7bfab1e0b1f92b9e"}
Nov 03 19:45:29.868874 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:29.868216    5600 generic.go:356] "Generic (PLEG): container finished" podID="db404f1b-c1b6-4b18-a5f6-16feb0de038a" containerID="cc3b556fc586d67d3280f67e414c32e4631b9b919498eda880633507e75eddd3" exitCode=0
Nov 03 19:45:29.868874 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:29.868272    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb" event={"ID":"db404f1b-c1b6-4b18-a5f6-16feb0de038a","Type":"ContainerDied","Data":"cc3b556fc586d67d3280f67e414c32e4631b9b919498eda880633507e75eddd3"}
Nov 03 19:45:29.873455 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:29.873344    5600 generic.go:356] "Generic (PLEG): container finished" podID="36298071-9373-4c44-b50f-32d783144cbf" containerID="70860eaec1e6f6e716a4d0ef803afbf0ccdba1c592a555807eb959e64b374963" exitCode=0
Nov 03 19:45:29.873455 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:29.873405    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv" event={"ID":"36298071-9373-4c44-b50f-32d783144cbf","Type":"ContainerDied","Data":"70860eaec1e6f6e716a4d0ef803afbf0ccdba1c592a555807eb959e64b374963"}
Nov 03 19:45:41.982313 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:41.981805    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d" event={"ID":"400d53f3-892f-4726-bc5c-8ecf1c503846","Type":"ContainerStarted","Data":"37a05b0557a1fb6575f340187db5cd81150d552fb8cae8f61fe61c09bd93cc92"}
Nov 03 19:45:41.992837 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:41.992805    5600 generic.go:356] "Generic (PLEG): container finished" podID="568b5e19-06b2-482d-877f-66704c6d1b0b" containerID="13807af4fbbb0ac7418af1dde6b1a94b28019191e13d7ba182ea3be82971f9ae" exitCode=0
Nov 03 19:45:41.992998 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:41.992891    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn" event={"ID":"568b5e19-06b2-482d-877f-66704c6d1b0b","Type":"ContainerDied","Data":"13807af4fbbb0ac7418af1dde6b1a94b28019191e13d7ba182ea3be82971f9ae"}
Nov 03 19:45:41.998196 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:41.997605    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj" event={"ID":"40b6a1b6-a798-46af-b02c-49314d794a52","Type":"ContainerStarted","Data":"a26765487a2c011875780b70867af36daa3e296396f48153bf7f028fb0ba2500"}
Nov 03 19:45:42.000301 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:42.000254    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d" podStartSLOduration=2.599216608 podStartE2EDuration="30.000236605s" totalImagesPullingTime="27.401019997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:41.999170714 +0000 UTC m=+324.511992497" watchObservedRunningTime="2026-11-03 19:45:42.000236605 +0000 UTC m=+324.513058371"
Nov 03 19:45:42.045194 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:42.045136    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj" podStartSLOduration=2.4916834039999998 podStartE2EDuration="28.04511608s" totalImagesPullingTime="25.553432676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:42.035559577 +0000 UTC m=+324.548381372" watchObservedRunningTime="2026-11-03 19:45:42.04511608 +0000 UTC m=+324.557937863"
Nov 03 19:45:43.004829 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.004789    5600 generic.go:356] "Generic (PLEG): container finished" podID="40b6a1b6-a798-46af-b02c-49314d794a52" containerID="a26765487a2c011875780b70867af36daa3e296396f48153bf7f028fb0ba2500" exitCode=0
Nov 03 19:45:43.005332 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.005300    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj" event={"ID":"40b6a1b6-a798-46af-b02c-49314d794a52","Type":"ContainerDied","Data":"a26765487a2c011875780b70867af36daa3e296396f48153bf7f028fb0ba2500"}
Nov 03 19:45:43.011138 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.011101    5600 generic.go:356] "Generic (PLEG): container finished" podID="16a6b515-bcb9-4bc4-8012-dc4ef0894da3" containerID="a9bd806c11970458b114dec8448ec962d55ba4aecaefc67277da7e10a607f64a" exitCode=0
Nov 03 19:45:43.014977 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.011519    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp" event={"ID":"16a6b515-bcb9-4bc4-8012-dc4ef0894da3","Type":"ContainerDied","Data":"a9bd806c11970458b114dec8448ec962d55ba4aecaefc67277da7e10a607f64a"}
Nov 03 19:45:43.014977 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.014775    5600 generic.go:356] "Generic (PLEG): container finished" podID="400d53f3-892f-4726-bc5c-8ecf1c503846" containerID="37a05b0557a1fb6575f340187db5cd81150d552fb8cae8f61fe61c09bd93cc92" exitCode=0
Nov 03 19:45:43.014977 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.014902    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d" event={"ID":"400d53f3-892f-4726-bc5c-8ecf1c503846","Type":"ContainerDied","Data":"37a05b0557a1fb6575f340187db5cd81150d552fb8cae8f61fe61c09bd93cc92"}
Nov 03 19:45:43.205277 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.205199    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:43.329991 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.329951    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568b5e19-06b2-482d-877f-66704c6d1b0b-kube-api-access-w2vf4\" (UniqueName: \"kubernetes.io/projected/568b5e19-06b2-482d-877f-66704c6d1b0b-kube-api-access-w2vf4\") pod \"568b5e19-06b2-482d-877f-66704c6d1b0b\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") "
Nov 03 19:45:43.330189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.330062    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-util\" (UniqueName: \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-util\") pod \"568b5e19-06b2-482d-877f-66704c6d1b0b\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") "
Nov 03 19:45:43.330189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.330119    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-bundle\") pod \"568b5e19-06b2-482d-877f-66704c6d1b0b\" (UID: \"568b5e19-06b2-482d-877f-66704c6d1b0b\") "
Nov 03 19:45:43.331189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.331158    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-bundle" pod "568b5e19-06b2-482d-877f-66704c6d1b0b" (UID: "568b5e19-06b2-482d-877f-66704c6d1b0b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:43.333699 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.333658    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568b5e19-06b2-482d-877f-66704c6d1b0b-kube-api-access-w2vf4" pod "568b5e19-06b2-482d-877f-66704c6d1b0b" (UID: "568b5e19-06b2-482d-877f-66704c6d1b0b"). InnerVolumeSpecName "kube-api-access-w2vf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:43.335275 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.335246    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-util" pod "568b5e19-06b2-482d-877f-66704c6d1b0b" (UID: "568b5e19-06b2-482d-877f-66704c6d1b0b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:43.431422 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.431374    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2vf4\" (UniqueName: \"kubernetes.io/projected/568b5e19-06b2-482d-877f-66704c6d1b0b-kube-api-access-w2vf4\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:43.431422 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.431416    5600 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-util\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:43.431422 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:43.431445    5600 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/568b5e19-06b2-482d-877f-66704c6d1b0b-bundle\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:44.022279 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.022217    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn"
Nov 03 19:45:44.022279 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.022257    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t2msn" event={"ID":"568b5e19-06b2-482d-877f-66704c6d1b0b","Type":"ContainerDied","Data":"819fd902f8569722fad6e76973c373a04c749172944350a1a862e7e183c945db"}
Nov 03 19:45:44.022838 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.022319    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="819fd902f8569722fad6e76973c373a04c749172944350a1a862e7e183c945db"
Nov 03 19:45:44.222100 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.222068    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:44.239648 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.239620    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:44.251764 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.251732    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:44.341670 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.341552    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-util\" (UniqueName: \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-util\") pod \"400d53f3-892f-4726-bc5c-8ecf1c503846\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") "
Nov 03 19:45:44.341670 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.341632    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-bundle\" (UniqueName: \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-bundle\") pod \"400d53f3-892f-4726-bc5c-8ecf1c503846\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") "
Nov 03 19:45:44.341899 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.341673    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400d53f3-892f-4726-bc5c-8ecf1c503846-kube-api-access-f2wql\" (UniqueName: \"kubernetes.io/projected/400d53f3-892f-4726-bc5c-8ecf1c503846-kube-api-access-f2wql\") pod \"400d53f3-892f-4726-bc5c-8ecf1c503846\" (UID: \"400d53f3-892f-4726-bc5c-8ecf1c503846\") "
Nov 03 19:45:44.341899 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.341715    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-util\" (UniqueName: \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-util\") pod \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") "
Nov 03 19:45:44.341899 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.341765    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-util\" (UniqueName: \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-util\") pod \"40b6a1b6-a798-46af-b02c-49314d794a52\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") "
Nov 03 19:45:44.341899 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.341823    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-bundle\" (UniqueName: \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-bundle\") pod \"40b6a1b6-a798-46af-b02c-49314d794a52\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") "
Nov 03 19:45:44.341899 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.341863    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b6a1b6-a798-46af-b02c-49314d794a52-kube-api-access-mst66\" (UniqueName: \"kubernetes.io/projected/40b6a1b6-a798-46af-b02c-49314d794a52-kube-api-access-mst66\") pod \"40b6a1b6-a798-46af-b02c-49314d794a52\" (UID: \"40b6a1b6-a798-46af-b02c-49314d794a52\") "
Nov 03 19:45:44.341899 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.341892    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-kube-api-access-fsns5\" (UniqueName: \"kubernetes.io/projected/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-kube-api-access-fsns5\") pod \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") "
Nov 03 19:45:44.342201 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.341919    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-bundle\") pod \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\" (UID: \"16a6b515-bcb9-4bc4-8012-dc4ef0894da3\") "
Nov 03 19:45:44.343137 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.343103    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-bundle" pod "16a6b515-bcb9-4bc4-8012-dc4ef0894da3" (UID: "16a6b515-bcb9-4bc4-8012-dc4ef0894da3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:44.343419 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.343394    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-bundle" pod "40b6a1b6-a798-46af-b02c-49314d794a52" (UID: "40b6a1b6-a798-46af-b02c-49314d794a52"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:44.347922 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.347892    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400d53f3-892f-4726-bc5c-8ecf1c503846-kube-api-access-f2wql" pod "400d53f3-892f-4726-bc5c-8ecf1c503846" (UID: "400d53f3-892f-4726-bc5c-8ecf1c503846"). InnerVolumeSpecName "kube-api-access-f2wql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:44.348055 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.348028    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-bundle" pod "400d53f3-892f-4726-bc5c-8ecf1c503846" (UID: "400d53f3-892f-4726-bc5c-8ecf1c503846"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:44.348143 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.348034    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-kube-api-access-fsns5" pod "16a6b515-bcb9-4bc4-8012-dc4ef0894da3" (UID: "16a6b515-bcb9-4bc4-8012-dc4ef0894da3"). InnerVolumeSpecName "kube-api-access-fsns5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:44.349348 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.349316    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b6a1b6-a798-46af-b02c-49314d794a52-kube-api-access-mst66" pod "40b6a1b6-a798-46af-b02c-49314d794a52" (UID: "40b6a1b6-a798-46af-b02c-49314d794a52"). InnerVolumeSpecName "kube-api-access-mst66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:44.354168 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.354142    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-util" pod "400d53f3-892f-4726-bc5c-8ecf1c503846" (UID: "400d53f3-892f-4726-bc5c-8ecf1c503846"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:44.354893 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.354871    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-util" pod "40b6a1b6-a798-46af-b02c-49314d794a52" (UID: "40b6a1b6-a798-46af-b02c-49314d794a52"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:44.355340 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.355316    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-util" pod "16a6b515-bcb9-4bc4-8012-dc4ef0894da3" (UID: "16a6b515-bcb9-4bc4-8012-dc4ef0894da3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:44.443369 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.443340    5600 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-util\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:44.443369 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.443371    5600 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40b6a1b6-a798-46af-b02c-49314d794a52-bundle\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:44.443564 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.443388    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mst66\" (UniqueName: \"kubernetes.io/projected/40b6a1b6-a798-46af-b02c-49314d794a52-kube-api-access-mst66\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:44.443564 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.443405    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsns5\" (UniqueName: \"kubernetes.io/projected/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-kube-api-access-fsns5\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:44.443564 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.443422    5600 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-bundle\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:44.443564 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.443457    5600 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-util\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:44.443564 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.443469    5600 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400d53f3-892f-4726-bc5c-8ecf1c503846-bundle\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:44.443564 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.443483    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2wql\" (UniqueName: \"kubernetes.io/projected/400d53f3-892f-4726-bc5c-8ecf1c503846-kube-api-access-f2wql\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:44.443564 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:44.443496    5600 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16a6b515-bcb9-4bc4-8012-dc4ef0894da3-util\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:45.028048 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.028021    5600 generic.go:356] "Generic (PLEG): container finished" podID="36298071-9373-4c44-b50f-32d783144cbf" containerID="95069a99c9cf6505d72af87298347d3059c3522b7908764a3ffda35fd439eb30" exitCode=0
Nov 03 19:45:45.028403 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.028109    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv" event={"ID":"36298071-9373-4c44-b50f-32d783144cbf","Type":"ContainerDied","Data":"95069a99c9cf6505d72af87298347d3059c3522b7908764a3ffda35fd439eb30"}
Nov 03 19:45:45.029901 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.029883    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp"
Nov 03 19:45:45.030006 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.029901    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qbtsp" event={"ID":"16a6b515-bcb9-4bc4-8012-dc4ef0894da3","Type":"ContainerDied","Data":"ad24ac0ecaeffa21b055f9ff3360886b133082d4e36868387211adde2ff6779e"}
Nov 03 19:45:45.030006 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.029936    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad24ac0ecaeffa21b055f9ff3360886b133082d4e36868387211adde2ff6779e"
Nov 03 19:45:45.031866 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.031838    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d"
Nov 03 19:45:45.031957 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.031842    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9nk5d" event={"ID":"400d53f3-892f-4726-bc5c-8ecf1c503846","Type":"ContainerDied","Data":"87e8c9207173bf508f1e7b52397642b782600f86b52d548f96ef3acdc1431490"}
Nov 03 19:45:45.031957 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.031952    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e8c9207173bf508f1e7b52397642b782600f86b52d548f96ef3acdc1431490"
Nov 03 19:45:45.034109 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.034090    5600 generic.go:356] "Generic (PLEG): container finished" podID="d74a273d-d3af-4738-97d0-f3bb2f193b5f" containerID="4dac03e43c6baa939eb1bfb56e4781030e4b92339eea5f115010dc7b2b21c34e" exitCode=0
Nov 03 19:45:45.034207 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.034158    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc" event={"ID":"d74a273d-d3af-4738-97d0-f3bb2f193b5f","Type":"ContainerDied","Data":"4dac03e43c6baa939eb1bfb56e4781030e4b92339eea5f115010dc7b2b21c34e"}
Nov 03 19:45:45.037080 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.037001    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj" event={"ID":"40b6a1b6-a798-46af-b02c-49314d794a52","Type":"ContainerDied","Data":"55bba11c47a62efe1e3c5cddb7e761f7a9aaa6c8ca9a3d2dea1728f3c4430663"}
Nov 03 19:45:45.037080 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.037034    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55bba11c47a62efe1e3c5cddb7e761f7a9aaa6c8ca9a3d2dea1728f3c4430663"
Nov 03 19:45:45.037200 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:45.037109    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvq6lj"
Nov 03 19:45:46.043625 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.043588    5600 generic.go:356] "Generic (PLEG): container finished" podID="db404f1b-c1b6-4b18-a5f6-16feb0de038a" containerID="00c8945a2702ea37bc57aaca6fb7ee7d66a427a43c7c9e59205d711ee0c04d5d" exitCode=0
Nov 03 19:45:46.044006 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.043674    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb" event={"ID":"db404f1b-c1b6-4b18-a5f6-16feb0de038a","Type":"ContainerDied","Data":"00c8945a2702ea37bc57aaca6fb7ee7d66a427a43c7c9e59205d711ee0c04d5d"}
Nov 03 19:45:46.208662 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.208639    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:46.237316 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.237290    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:46.362326 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.362246    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36298071-9373-4c44-b50f-32d783144cbf-kube-api-access-wnbxg\" (UniqueName: \"kubernetes.io/projected/36298071-9373-4c44-b50f-32d783144cbf-kube-api-access-wnbxg\") pod \"36298071-9373-4c44-b50f-32d783144cbf\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") "
Nov 03 19:45:46.362326 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.362291    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-util\" (UniqueName: \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-util\") pod \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") "
Nov 03 19:45:46.362564 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.362335    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-bundle\") pod \"36298071-9373-4c44-b50f-32d783144cbf\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") "
Nov 03 19:45:46.362564 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.362368    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-bundle\") pod \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") "
Nov 03 19:45:46.362564 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.362454    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-util\" (UniqueName: \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-util\") pod \"36298071-9373-4c44-b50f-32d783144cbf\" (UID: \"36298071-9373-4c44-b50f-32d783144cbf\") "
Nov 03 19:45:46.362564 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.362512    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74a273d-d3af-4738-97d0-f3bb2f193b5f-kube-api-access-zqk4z\" (UniqueName: \"kubernetes.io/projected/d74a273d-d3af-4738-97d0-f3bb2f193b5f-kube-api-access-zqk4z\") pod \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\" (UID: \"d74a273d-d3af-4738-97d0-f3bb2f193b5f\") "
Nov 03 19:45:46.362892 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.362866    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-bundle" pod "d74a273d-d3af-4738-97d0-f3bb2f193b5f" (UID: "d74a273d-d3af-4738-97d0-f3bb2f193b5f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:46.363332 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.363187    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-bundle" pod "36298071-9373-4c44-b50f-32d783144cbf" (UID: "36298071-9373-4c44-b50f-32d783144cbf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:46.364777 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.364750    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74a273d-d3af-4738-97d0-f3bb2f193b5f-kube-api-access-zqk4z" pod "d74a273d-d3af-4738-97d0-f3bb2f193b5f" (UID: "d74a273d-d3af-4738-97d0-f3bb2f193b5f"). InnerVolumeSpecName "kube-api-access-zqk4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:46.364899 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.364781    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36298071-9373-4c44-b50f-32d783144cbf-kube-api-access-wnbxg" pod "36298071-9373-4c44-b50f-32d783144cbf" (UID: "36298071-9373-4c44-b50f-32d783144cbf"). InnerVolumeSpecName "kube-api-access-wnbxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:46.368315 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.368291    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-util" pod "d74a273d-d3af-4738-97d0-f3bb2f193b5f" (UID: "d74a273d-d3af-4738-97d0-f3bb2f193b5f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:46.368790 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.368562    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-util" pod "36298071-9373-4c44-b50f-32d783144cbf" (UID: "36298071-9373-4c44-b50f-32d783144cbf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:46.463259 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.463226    5600 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-bundle\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:46.463259 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.463253    5600 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-bundle\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:46.463259 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.463262    5600 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36298071-9373-4c44-b50f-32d783144cbf-util\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:46.463495 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.463271    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqk4z\" (UniqueName: \"kubernetes.io/projected/d74a273d-d3af-4738-97d0-f3bb2f193b5f-kube-api-access-zqk4z\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:46.463495 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.463284    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbxg\" (UniqueName: \"kubernetes.io/projected/36298071-9373-4c44-b50f-32d783144cbf-kube-api-access-wnbxg\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:46.463495 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:46.463306    5600 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d74a273d-d3af-4738-97d0-f3bb2f193b5f-util\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:47.049797 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.049763    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc"
Nov 03 19:45:47.049797 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.049783    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644srzqc" event={"ID":"d74a273d-d3af-4738-97d0-f3bb2f193b5f","Type":"ContainerDied","Data":"fc301d9850898324493bda471ff20192e7cbe251d159cca38099cb6eb146b124"}
Nov 03 19:45:47.050482 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.049840    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc301d9850898324493bda471ff20192e7cbe251d159cca38099cb6eb146b124"
Nov 03 19:45:47.051724 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.051706    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv"
Nov 03 19:45:47.051862 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.051736    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aptpkv" event={"ID":"36298071-9373-4c44-b50f-32d783144cbf","Type":"ContainerDied","Data":"aa6b4d02246d95b2d4af4e07bdf6fa3b4557569790ba013c4d100d66d36e7be7"}
Nov 03 19:45:47.051862 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.051766    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa6b4d02246d95b2d4af4e07bdf6fa3b4557569790ba013c4d100d66d36e7be7"
Nov 03 19:45:47.180025 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.179995    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:47.268555 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.268524    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-bundle\") pod \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") "
Nov 03 19:45:47.268555 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.268565    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db404f1b-c1b6-4b18-a5f6-16feb0de038a-kube-api-access-8q9f7\" (UniqueName: \"kubernetes.io/projected/db404f1b-c1b6-4b18-a5f6-16feb0de038a-kube-api-access-8q9f7\") pod \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") "
Nov 03 19:45:47.268817 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.268593    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-util\" (UniqueName: \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-util\") pod \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\" (UID: \"db404f1b-c1b6-4b18-a5f6-16feb0de038a\") "
Nov 03 19:45:47.269108 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.269085    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-bundle" pod "db404f1b-c1b6-4b18-a5f6-16feb0de038a" (UID: "db404f1b-c1b6-4b18-a5f6-16feb0de038a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:47.270903 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.270856    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db404f1b-c1b6-4b18-a5f6-16feb0de038a-kube-api-access-8q9f7" pod "db404f1b-c1b6-4b18-a5f6-16feb0de038a" (UID: "db404f1b-c1b6-4b18-a5f6-16feb0de038a"). InnerVolumeSpecName "kube-api-access-8q9f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:47.273877 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.273857    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-util" pod "db404f1b-c1b6-4b18-a5f6-16feb0de038a" (UID: "db404f1b-c1b6-4b18-a5f6-16feb0de038a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:47.369819 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.369735    5600 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-bundle\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:47.369819 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.369769    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q9f7\" (UniqueName: \"kubernetes.io/projected/db404f1b-c1b6-4b18-a5f6-16feb0de038a-kube-api-access-8q9f7\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:47.369819 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:47.369780    5600 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db404f1b-c1b6-4b18-a5f6-16feb0de038a-util\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:48.057904 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.057872    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb" event={"ID":"db404f1b-c1b6-4b18-a5f6-16feb0de038a","Type":"ContainerDied","Data":"c2fadf8bef49aff62d30dd9d1efe4ae47db50eb7587d897d2ff21ef33013a57e"}
Nov 03 19:45:48.058288 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.057910    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79j8kb"
Nov 03 19:45:48.058288 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.057918    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2fadf8bef49aff62d30dd9d1efe4ae47db50eb7587d897d2ff21ef33013a57e"
Nov 03 19:45:48.582872 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.582840    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"]
Nov 03 19:45:48.583160 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583147    5600 cpu_manager.go:414] "containerMap: removing container" podUID="40b6a1b6-a798-46af-b02c-49314d794a52" containerName="pull"
Nov 03 19:45:48.583205 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583162    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b6a1b6-a798-46af-b02c-49314d794a52" containerName="pull"
Nov 03 19:45:48.583205 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583173    5600 cpu_manager.go:414] "containerMap: removing container" podUID="db404f1b-c1b6-4b18-a5f6-16feb0de038a" containerName="pull"
Nov 03 19:45:48.583205 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583178    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db404f1b-c1b6-4b18-a5f6-16feb0de038a" containerName="pull"
Nov 03 19:45:48.583205 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583189    5600 cpu_manager.go:414] "containerMap: removing container" podUID="16a6b515-bcb9-4bc4-8012-dc4ef0894da3" containerName="extract"
Nov 03 19:45:48.583205 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583194    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a6b515-bcb9-4bc4-8012-dc4ef0894da3" containerName="extract"
Nov 03 19:45:48.583205 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583201    5600 cpu_manager.go:414] "containerMap: removing container" podUID="db404f1b-c1b6-4b18-a5f6-16feb0de038a" containerName="extract"
Nov 03 19:45:48.583205 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583206    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db404f1b-c1b6-4b18-a5f6-16feb0de038a" containerName="extract"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583213    5600 cpu_manager.go:414] "containerMap: removing container" podUID="400d53f3-892f-4726-bc5c-8ecf1c503846" containerName="pull"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583217    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400d53f3-892f-4726-bc5c-8ecf1c503846" containerName="pull"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583224    5600 cpu_manager.go:414] "containerMap: removing container" podUID="40b6a1b6-a798-46af-b02c-49314d794a52" containerName="extract"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583228    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b6a1b6-a798-46af-b02c-49314d794a52" containerName="extract"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583238    5600 cpu_manager.go:414] "containerMap: removing container" podUID="d74a273d-d3af-4738-97d0-f3bb2f193b5f" containerName="pull"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583243    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74a273d-d3af-4738-97d0-f3bb2f193b5f" containerName="pull"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583250    5600 cpu_manager.go:414] "containerMap: removing container" podUID="568b5e19-06b2-482d-877f-66704c6d1b0b" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583255    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568b5e19-06b2-482d-877f-66704c6d1b0b" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583275    5600 cpu_manager.go:414] "containerMap: removing container" podUID="16a6b515-bcb9-4bc4-8012-dc4ef0894da3" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583280    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a6b515-bcb9-4bc4-8012-dc4ef0894da3" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583286    5600 cpu_manager.go:414] "containerMap: removing container" podUID="568b5e19-06b2-482d-877f-66704c6d1b0b" containerName="pull"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583290    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568b5e19-06b2-482d-877f-66704c6d1b0b" containerName="pull"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583295    5600 cpu_manager.go:414] "containerMap: removing container" podUID="568b5e19-06b2-482d-877f-66704c6d1b0b" containerName="extract"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583300    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568b5e19-06b2-482d-877f-66704c6d1b0b" containerName="extract"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583305    5600 cpu_manager.go:414] "containerMap: removing container" podUID="d74a273d-d3af-4738-97d0-f3bb2f193b5f" containerName="extract"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583309    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74a273d-d3af-4738-97d0-f3bb2f193b5f" containerName="extract"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583317    5600 cpu_manager.go:414] "containerMap: removing container" podUID="400d53f3-892f-4726-bc5c-8ecf1c503846" containerName="extract"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583320    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400d53f3-892f-4726-bc5c-8ecf1c503846" containerName="extract"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583329    5600 cpu_manager.go:414] "containerMap: removing container" podUID="16a6b515-bcb9-4bc4-8012-dc4ef0894da3" containerName="pull"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583333    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a6b515-bcb9-4bc4-8012-dc4ef0894da3" containerName="pull"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583341    5600 cpu_manager.go:414] "containerMap: removing container" podUID="40b6a1b6-a798-46af-b02c-49314d794a52" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583345    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b6a1b6-a798-46af-b02c-49314d794a52" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583352    5600 cpu_manager.go:414] "containerMap: removing container" podUID="36298071-9373-4c44-b50f-32d783144cbf" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583356    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36298071-9373-4c44-b50f-32d783144cbf" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583364    5600 cpu_manager.go:414] "containerMap: removing container" podUID="d74a273d-d3af-4738-97d0-f3bb2f193b5f" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583368    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74a273d-d3af-4738-97d0-f3bb2f193b5f" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583373    5600 cpu_manager.go:414] "containerMap: removing container" podUID="400d53f3-892f-4726-bc5c-8ecf1c503846" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583377    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400d53f3-892f-4726-bc5c-8ecf1c503846" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583383    5600 cpu_manager.go:414] "containerMap: removing container" podUID="36298071-9373-4c44-b50f-32d783144cbf" containerName="pull"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583387    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36298071-9373-4c44-b50f-32d783144cbf" containerName="pull"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583394    5600 cpu_manager.go:414] "containerMap: removing container" podUID="36298071-9373-4c44-b50f-32d783144cbf" containerName="extract"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583397    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36298071-9373-4c44-b50f-32d783144cbf" containerName="extract"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583403    5600 cpu_manager.go:414] "containerMap: removing container" podUID="db404f1b-c1b6-4b18-a5f6-16feb0de038a" containerName="util"
Nov 03 19:45:48.583408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583407    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db404f1b-c1b6-4b18-a5f6-16feb0de038a" containerName="util"
Nov 03 19:45:48.584345 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583465    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="36298071-9373-4c44-b50f-32d783144cbf" containerName="extract"
Nov 03 19:45:48.584345 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583473    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a6b515-bcb9-4bc4-8012-dc4ef0894da3" containerName="extract"
Nov 03 19:45:48.584345 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583478    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="db404f1b-c1b6-4b18-a5f6-16feb0de038a" containerName="extract"
Nov 03 19:45:48.584345 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583483    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b6a1b6-a798-46af-b02c-49314d794a52" containerName="extract"
Nov 03 19:45:48.584345 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583490    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74a273d-d3af-4738-97d0-f3bb2f193b5f" containerName="extract"
Nov 03 19:45:48.584345 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583495    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="568b5e19-06b2-482d-877f-66704c6d1b0b" containerName="extract"
Nov 03 19:45:48.584345 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.583500    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="400d53f3-892f-4726-bc5c-8ecf1c503846" containerName="extract"
Nov 03 19:45:48.586689 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.586672    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.597680 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.597660    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 19:45:48.597830 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.597815    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 19:45:48.597925 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.597910    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 19:45:48.597991 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.597950    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 19:45:48.597991 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.597980    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 19:45:48.598150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.598132    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-85znx\""
Nov 03 19:45:48.627983 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.627960    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"]
Nov 03 19:45:48.682150 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.682119    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/64b489b3-2516-451a-9c2d-95914c2bed5c-metrics-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.682268 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.682152    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64b489b3-2516-451a-9c2d-95914c2bed5c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.682268 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.682173    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkpvl\" (UniqueName: \"kubernetes.io/projected/64b489b3-2516-451a-9c2d-95914c2bed5c-kube-api-access-hkpvl\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.682268 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.682192    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64b489b3-2516-451a-9c2d-95914c2bed5c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.682415 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.682392    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64b489b3-2516-451a-9c2d-95914c2bed5c-manifests\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.682483 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.682457    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64b489b3-2516-451a-9c2d-95914c2bed5c-tmp\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.783569 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.783533    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/64b489b3-2516-451a-9c2d-95914c2bed5c-metrics-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.783720 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.783593    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64b489b3-2516-451a-9c2d-95914c2bed5c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.783720 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.783612    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkpvl\" (UniqueName: \"kubernetes.io/projected/64b489b3-2516-451a-9c2d-95914c2bed5c-kube-api-access-hkpvl\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.783720 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.783631    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64b489b3-2516-451a-9c2d-95914c2bed5c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.783720 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.783695    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64b489b3-2516-451a-9c2d-95914c2bed5c-manifests\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.783908 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.783722    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64b489b3-2516-451a-9c2d-95914c2bed5c-tmp\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.784212 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.784180    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64b489b3-2516-451a-9c2d-95914c2bed5c-manifests\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.786102 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.786080    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/64b489b3-2516-451a-9c2d-95914c2bed5c-metrics-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.786213 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.786196    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64b489b3-2516-451a-9c2d-95914c2bed5c-tmp\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.786334 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.786312    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64b489b3-2516-451a-9c2d-95914c2bed5c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.786380 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.786328    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64b489b3-2516-451a-9c2d-95914c2bed5c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.791546 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.791523    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkpvl\" (UniqueName: \"kubernetes.io/projected/64b489b3-2516-451a-9c2d-95914c2bed5c-kube-api-access-hkpvl\") pod \"opendatahub-operator-controller-manager-5749d45fbb-lbhkn\" (UID: \"64b489b3-2516-451a-9c2d-95914c2bed5c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:48.906152 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:48.906119    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:49.196925 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:49.196898    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"]
Nov 03 19:45:49.198741 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:45:49.198716    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b489b3_2516_451a_9c2d_95914c2bed5c.slice/crio-45708ab77318ffc47a04de7cdcfc26e4a72f1c359edd80fa274a9601f7341ace WatchSource:0}: Error finding container 45708ab77318ffc47a04de7cdcfc26e4a72f1c359edd80fa274a9601f7341ace: Status 404 returned error can't find the container with id 45708ab77318ffc47a04de7cdcfc26e4a72f1c359edd80fa274a9601f7341ace
Nov 03 19:45:50.069260 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:50.069224    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn" event={"ID":"64b489b3-2516-451a-9c2d-95914c2bed5c","Type":"ContainerStarted","Data":"45708ab77318ffc47a04de7cdcfc26e4a72f1c359edd80fa274a9601f7341ace"}
Nov 03 19:45:55.091478 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:55.091413    5600 generic.go:356] "Generic (PLEG): container finished" podID="64b489b3-2516-451a-9c2d-95914c2bed5c" containerID="6a9ad20d04195f04a73eb9a012281bbaca1d71acbed6031686cdcc607fb87864" exitCode=0
Nov 03 19:45:55.091856 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:55.091504    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn" event={"ID":"64b489b3-2516-451a-9c2d-95914c2bed5c","Type":"ContainerDied","Data":"6a9ad20d04195f04a73eb9a012281bbaca1d71acbed6031686cdcc607fb87864"}
Nov 03 19:45:56.098226 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:56.098192    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn" event={"ID":"64b489b3-2516-451a-9c2d-95914c2bed5c","Type":"ContainerStarted","Data":"34cc359dfdac3e0f49f4ce061a93605296591d970f261473df0dd24d800622d5"}
Nov 03 19:45:56.098703 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:56.098344    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:45:56.145315 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:56.145264    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn" podStartSLOduration=1.998787316 podStartE2EDuration="8.145248215s" totalImagesPullingTime="5.729032693s" totalInitContainerRuntime="417.428206ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:56.14338902 +0000 UTC m=+338.656210801" watchObservedRunningTime="2026-11-03 19:45:56.145248215 +0000 UTC m=+338.658069998"
Nov 03 19:45:58.796885 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:45:58.796841    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:45:59.872505 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:59.872478    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8"]
Nov 03 19:45:59.877478 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:59.877454    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8"
Nov 03 19:45:59.881026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:59.881002    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvgf\" (UniqueName: \"kubernetes.io/projected/60ca69b0-824e-4a96-a238-7d5416576c66-kube-api-access-brvgf\") pod \"authorino-operator-7b7bb4bc77-2sjs8\" (UID: \"60ca69b0-824e-4a96-a238-7d5416576c66\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8"
Nov 03 19:45:59.885696 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:59.885663    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 19:45:59.885809 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:59.885748    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-slw59\""
Nov 03 19:45:59.885809 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:59.885779    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 19:45:59.887839 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:59.887819    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8"]
Nov 03 19:45:59.981357 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:59.981319    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvgf\" (UniqueName: \"kubernetes.io/projected/60ca69b0-824e-4a96-a238-7d5416576c66-kube-api-access-brvgf\") pod \"authorino-operator-7b7bb4bc77-2sjs8\" (UID: \"60ca69b0-824e-4a96-a238-7d5416576c66\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8"
Nov 03 19:45:59.989606 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:45:59.989578    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvgf\" (UniqueName: \"kubernetes.io/projected/60ca69b0-824e-4a96-a238-7d5416576c66-kube-api-access-brvgf\") pod \"authorino-operator-7b7bb4bc77-2sjs8\" (UID: \"60ca69b0-824e-4a96-a238-7d5416576c66\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8"
Nov 03 19:46:00.190771 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:00.190731    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8"
Nov 03 19:46:00.463003 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:00.462920    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8"]
Nov 03 19:46:00.465151 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:46:00.465122    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ca69b0_824e_4a96_a238_7d5416576c66.slice/crio-ba38a2fa9f4a2a1a09d3e68b80d28e62c5925e1f6ef18070f4f3793816850754 WatchSource:0}: Error finding container ba38a2fa9f4a2a1a09d3e68b80d28e62c5925e1f6ef18070f4f3793816850754: Status 404 returned error can't find the container with id ba38a2fa9f4a2a1a09d3e68b80d28e62c5925e1f6ef18070f4f3793816850754
Nov 03 19:46:01.121678 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.121643    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8" event={"ID":"60ca69b0-824e-4a96-a238-7d5416576c66","Type":"ContainerStarted","Data":"ba38a2fa9f4a2a1a09d3e68b80d28e62c5925e1f6ef18070f4f3793816850754"}
Nov 03 19:46:01.763514 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.763471    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658"]
Nov 03 19:46:01.769407 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.767514    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658"
Nov 03 19:46:01.778843 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.778815    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:46:01.778961 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.778923    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 19:46:01.778961 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.778946    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2r8lc\""
Nov 03 19:46:01.793332 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.793308    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658"]
Nov 03 19:46:01.795770 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.795744    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54e56fa2-fff2-42b8-b677-df084ae2187c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67658\" (UID: \"54e56fa2-fff2-42b8-b677-df084ae2187c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658"
Nov 03 19:46:01.795909 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.795890    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52qsk\" (UniqueName: \"kubernetes.io/projected/54e56fa2-fff2-42b8-b677-df084ae2187c-kube-api-access-52qsk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67658\" (UID: \"54e56fa2-fff2-42b8-b677-df084ae2187c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658"
Nov 03 19:46:01.896677 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.896641    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52qsk\" (UniqueName: \"kubernetes.io/projected/54e56fa2-fff2-42b8-b677-df084ae2187c-kube-api-access-52qsk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67658\" (UID: \"54e56fa2-fff2-42b8-b677-df084ae2187c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658"
Nov 03 19:46:01.896852 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.896707    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54e56fa2-fff2-42b8-b677-df084ae2187c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67658\" (UID: \"54e56fa2-fff2-42b8-b677-df084ae2187c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658"
Nov 03 19:46:01.897185 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.897164    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54e56fa2-fff2-42b8-b677-df084ae2187c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67658\" (UID: \"54e56fa2-fff2-42b8-b677-df084ae2187c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658"
Nov 03 19:46:01.905341 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:01.905311    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52qsk\" (UniqueName: \"kubernetes.io/projected/54e56fa2-fff2-42b8-b677-df084ae2187c-kube-api-access-52qsk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67658\" (UID: \"54e56fa2-fff2-42b8-b677-df084ae2187c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658"
Nov 03 19:46:02.083457 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:02.083363    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658"
Nov 03 19:46:02.361635 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:02.361572    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658"]
Nov 03 19:46:02.364505 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:46:02.364406    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e56fa2_fff2_42b8_b677_df084ae2187c.slice/crio-f5e3ca3926150668ba63f11af2f1bac23bb6bd26acc0dd8cb6aa61fc455a1a6c WatchSource:0}: Error finding container f5e3ca3926150668ba63f11af2f1bac23bb6bd26acc0dd8cb6aa61fc455a1a6c: Status 404 returned error can't find the container with id f5e3ca3926150668ba63f11af2f1bac23bb6bd26acc0dd8cb6aa61fc455a1a6c
Nov 03 19:46:03.133665 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:03.133628    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658" event={"ID":"54e56fa2-fff2-42b8-b677-df084ae2187c","Type":"ContainerStarted","Data":"f5e3ca3926150668ba63f11af2f1bac23bb6bd26acc0dd8cb6aa61fc455a1a6c"}
Nov 03 19:46:04.140799 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.140758    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8" event={"ID":"60ca69b0-824e-4a96-a238-7d5416576c66","Type":"ContainerStarted","Data":"586e5fa88b298baad6dbd1692c576869e690bf8387576ba3da425d27eaacb113"}
Nov 03 19:46:04.141249 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.140841    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8"
Nov 03 19:46:04.156048 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.156020    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d"]
Nov 03 19:46:04.159282 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.159264    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d"
Nov 03 19:46:04.161846 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.161804    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8" podStartSLOduration=2.195840374 podStartE2EDuration="5.161790977s" totalImagesPullingTime="2.965950603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:04.160545794 +0000 UTC m=+346.673367574" watchObservedRunningTime="2026-11-03 19:46:04.161790977 +0000 UTC m=+346.674612756"
Nov 03 19:46:04.167265 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.167216    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-x2s6j\""
Nov 03 19:46:04.167265 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.167233    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 19:46:04.172420 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.172394    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d"]
Nov 03 19:46:04.215161 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.215118    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxh8p\" (UniqueName: \"kubernetes.io/projected/de1f93a5-a1d4-4a1d-945c-f08aaa2b932a-kube-api-access-xxh8p\") pod \"dns-operator-controller-manager-8557cbdb84-bk89d\" (UID: \"de1f93a5-a1d4-4a1d-945c-f08aaa2b932a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d"
Nov 03 19:46:04.315882 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.315842    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxh8p\" (UniqueName: \"kubernetes.io/projected/de1f93a5-a1d4-4a1d-945c-f08aaa2b932a-kube-api-access-xxh8p\") pod \"dns-operator-controller-manager-8557cbdb84-bk89d\" (UID: \"de1f93a5-a1d4-4a1d-945c-f08aaa2b932a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d"
Nov 03 19:46:04.333189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.333157    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxh8p\" (UniqueName: \"kubernetes.io/projected/de1f93a5-a1d4-4a1d-945c-f08aaa2b932a-kube-api-access-xxh8p\") pod \"dns-operator-controller-manager-8557cbdb84-bk89d\" (UID: \"de1f93a5-a1d4-4a1d-945c-f08aaa2b932a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d"
Nov 03 19:46:04.476407 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.476318    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d"
Nov 03 19:46:04.775285 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:04.775211    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d"]
Nov 03 19:46:05.147598 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:05.147556    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658" event={"ID":"54e56fa2-fff2-42b8-b677-df084ae2187c","Type":"ContainerStarted","Data":"73b6880ddcbb9b043a97ba6060fec578ced4de335b9c421bb2beb4c85cf6194e"}
Nov 03 19:46:05.149112 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:05.149068    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d" event={"ID":"de1f93a5-a1d4-4a1d-945c-f08aaa2b932a","Type":"ContainerStarted","Data":"2e0a45cb749efee84a741553a5e70768c37671e14990b07253051e5b1d6915d3"}
Nov 03 19:46:05.175487 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:05.175412    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67658" podStartSLOduration=1.899458746 podStartE2EDuration="4.175395482s" totalImagesPullingTime="2.275936736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:05.173865488 +0000 UTC m=+347.686687270" watchObservedRunningTime="2026-11-03 19:46:05.175395482 +0000 UTC m=+347.688217268"
Nov 03 19:46:06.585504 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.585469    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"]
Nov 03 19:46:06.589940 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.589901    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"
Nov 03 19:46:06.606579 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.606478    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"]
Nov 03 19:46:06.606579 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.606531    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ktt8m\""
Nov 03 19:46:06.606579 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.606548    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 19:46:06.606934 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.606910    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 19:46:06.636023 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.635989    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adbf5e48-bc06-4df3-aa7d-fe2731c194f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-dq9xl\" (UID: \"adbf5e48-bc06-4df3-aa7d-fe2731c194f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"
Nov 03 19:46:06.636183 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.636055    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq9z8\" (UniqueName: \"kubernetes.io/projected/adbf5e48-bc06-4df3-aa7d-fe2731c194f8-kube-api-access-rq9z8\") pod \"cert-manager-webhook-5dbbbf75cc-dq9xl\" (UID: \"adbf5e48-bc06-4df3-aa7d-fe2731c194f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"
Nov 03 19:46:06.737276 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.737244    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adbf5e48-bc06-4df3-aa7d-fe2731c194f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-dq9xl\" (UID: \"adbf5e48-bc06-4df3-aa7d-fe2731c194f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"
Nov 03 19:46:06.737422 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.737290    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq9z8\" (UniqueName: \"kubernetes.io/projected/adbf5e48-bc06-4df3-aa7d-fe2731c194f8-kube-api-access-rq9z8\") pod \"cert-manager-webhook-5dbbbf75cc-dq9xl\" (UID: \"adbf5e48-bc06-4df3-aa7d-fe2731c194f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"
Nov 03 19:46:06.759054 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.759028    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq9z8\" (UniqueName: \"kubernetes.io/projected/adbf5e48-bc06-4df3-aa7d-fe2731c194f8-kube-api-access-rq9z8\") pod \"cert-manager-webhook-5dbbbf75cc-dq9xl\" (UID: \"adbf5e48-bc06-4df3-aa7d-fe2731c194f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"
Nov 03 19:46:06.759208 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.759082    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/adbf5e48-bc06-4df3-aa7d-fe2731c194f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-dq9xl\" (UID: \"adbf5e48-bc06-4df3-aa7d-fe2731c194f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"
Nov 03 19:46:06.914359 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:06.914329    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"
Nov 03 19:46:07.105636 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:07.105605    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-lbhkn"
Nov 03 19:46:07.192661 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:07.192596    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"]
Nov 03 19:46:07.196270 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:46:07.196244    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadbf5e48_bc06_4df3_aa7d_fe2731c194f8.slice/crio-87d5c91597915fc27e2d9587ccfa4b98cf707eb7d1b0163c8e0478e5c4383663 WatchSource:0}: Error finding container 87d5c91597915fc27e2d9587ccfa4b98cf707eb7d1b0163c8e0478e5c4383663: Status 404 returned error can't find the container with id 87d5c91597915fc27e2d9587ccfa4b98cf707eb7d1b0163c8e0478e5c4383663
Nov 03 19:46:08.164882 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:08.164828    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d" event={"ID":"de1f93a5-a1d4-4a1d-945c-f08aaa2b932a","Type":"ContainerStarted","Data":"8f8e0d2f86dd537c23c252b554597e6b42b2e0893bf3a144e9476d8f0bc622d7"}
Nov 03 19:46:08.165373 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:08.164938    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d"
Nov 03 19:46:08.166027 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:08.166005    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl" event={"ID":"adbf5e48-bc06-4df3-aa7d-fe2731c194f8","Type":"ContainerStarted","Data":"87d5c91597915fc27e2d9587ccfa4b98cf707eb7d1b0163c8e0478e5c4383663"}
Nov 03 19:46:08.187878 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:08.187832    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d" podStartSLOduration=1.59878482 podStartE2EDuration="4.187820545s" totalImagesPullingTime="2.589035725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:08.186022811 +0000 UTC m=+350.698844592" watchObservedRunningTime="2026-11-03 19:46:08.187820545 +0000 UTC m=+350.700642325"
Nov 03 19:46:11.181551 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.181517    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl" event={"ID":"adbf5e48-bc06-4df3-aa7d-fe2731c194f8","Type":"ContainerStarted","Data":"aebd45227bb3f7030333407c380604b7e343117b337a933d22261826eef1165f"}
Nov 03 19:46:11.181936 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.181759    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"
Nov 03 19:46:11.200102 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.200056    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl" podStartSLOduration=1.7883491390000001 podStartE2EDuration="5.200042947s" totalImagesPullingTime="3.411693808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:11.198339525 +0000 UTC m=+353.711161287" watchObservedRunningTime="2026-11-03 19:46:11.200042947 +0000 UTC m=+353.712864722"
Nov 03 19:46:11.458691 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.458608    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"]
Nov 03 19:46:11.463189 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.463163    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:46:11.478361 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.478332    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6m94\" (UniqueName: \"kubernetes.io/projected/b4a89640-183b-456f-8f5a-5307275dd08c-kube-api-access-r6m94\") pod \"kuadrant-operator-controller-manager-678d87766f-zwbxr\" (UID: \"b4a89640-183b-456f-8f5a-5307275dd08c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:46:11.478506 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.478494    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a89640-183b-456f-8f5a-5307275dd08c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zwbxr\" (UID: \"b4a89640-183b-456f-8f5a-5307275dd08c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:46:11.478867 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.478850    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d5jrn\""
Nov 03 19:46:11.480668 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.480641    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"]
Nov 03 19:46:11.579516 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.579475    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6m94\" (UniqueName: \"kubernetes.io/projected/b4a89640-183b-456f-8f5a-5307275dd08c-kube-api-access-r6m94\") pod \"kuadrant-operator-controller-manager-678d87766f-zwbxr\" (UID: \"b4a89640-183b-456f-8f5a-5307275dd08c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:46:11.579691 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.579582    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a89640-183b-456f-8f5a-5307275dd08c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zwbxr\" (UID: \"b4a89640-183b-456f-8f5a-5307275dd08c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:46:11.579994 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.579974    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a89640-183b-456f-8f5a-5307275dd08c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zwbxr\" (UID: \"b4a89640-183b-456f-8f5a-5307275dd08c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:46:11.589538 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.589507    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6m94\" (UniqueName: \"kubernetes.io/projected/b4a89640-183b-456f-8f5a-5307275dd08c-kube-api-access-r6m94\") pod \"kuadrant-operator-controller-manager-678d87766f-zwbxr\" (UID: \"b4a89640-183b-456f-8f5a-5307275dd08c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:46:11.775364 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:11.775278    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:46:12.071385 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:12.071362    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"]
Nov 03 19:46:12.072724 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:46:12.072701    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a89640_183b_456f_8f5a_5307275dd08c.slice/crio-531f5ea0da8b15089c4e3fd92b324f630b0049c1897376efbe5cf96328be14d3 WatchSource:0}: Error finding container 531f5ea0da8b15089c4e3fd92b324f630b0049c1897376efbe5cf96328be14d3: Status 404 returned error can't find the container with id 531f5ea0da8b15089c4e3fd92b324f630b0049c1897376efbe5cf96328be14d3
Nov 03 19:46:12.186919 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:12.186882    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr" event={"ID":"b4a89640-183b-456f-8f5a-5307275dd08c","Type":"ContainerStarted","Data":"531f5ea0da8b15089c4e3fd92b324f630b0049c1897376efbe5cf96328be14d3"}
Nov 03 19:46:15.151195 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:15.151165    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2sjs8"
Nov 03 19:46:17.189148 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:17.189118    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl"
Nov 03 19:46:18.219577 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:18.219542    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr" event={"ID":"b4a89640-183b-456f-8f5a-5307275dd08c","Type":"ContainerStarted","Data":"56932ee357a049c61aca0e508d9d797ed6d80e0524ea009c21336a3ad8721564"}
Nov 03 19:46:18.219977 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:18.219656    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:46:18.240705 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:18.240651    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr" podStartSLOduration=2.067334637 podStartE2EDuration="7.240639544s" totalImagesPullingTime="5.173304907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:18.2389567 +0000 UTC m=+360.751778492" watchObservedRunningTime="2026-11-03 19:46:18.240639544 +0000 UTC m=+360.753461375"
Nov 03 19:46:19.172868 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.172835    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bk89d"
Nov 03 19:46:19.424366 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.424291    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"]
Nov 03 19:46:19.429338 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.429314    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"
Nov 03 19:46:19.439397 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.439368    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 19:46:19.439533 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.439475    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 19:46:19.439607 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.439593    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 19:46:19.439679 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.439668    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-m9cc9\""
Nov 03 19:46:19.439779 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.439756    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 19:46:19.440290 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.440269    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"]
Nov 03 19:46:19.545352 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.545317    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6w4g\" (UniqueName: \"kubernetes.io/projected/20362046-2f57-4bea-8b2d-4a9092f60059-kube-api-access-c6w4g\") pod \"kube-auth-proxy-6dd88dbbb7-48gc7\" (UID: \"20362046-2f57-4bea-8b2d-4a9092f60059\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"
Nov 03 19:46:19.545549 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.545376    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20362046-2f57-4bea-8b2d-4a9092f60059-tls-certs\") pod \"kube-auth-proxy-6dd88dbbb7-48gc7\" (UID: \"20362046-2f57-4bea-8b2d-4a9092f60059\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"
Nov 03 19:46:19.545549 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.545483    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20362046-2f57-4bea-8b2d-4a9092f60059-tmp\") pod \"kube-auth-proxy-6dd88dbbb7-48gc7\" (UID: \"20362046-2f57-4bea-8b2d-4a9092f60059\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"
Nov 03 19:46:19.646408 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.646375    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6w4g\" (UniqueName: \"kubernetes.io/projected/20362046-2f57-4bea-8b2d-4a9092f60059-kube-api-access-c6w4g\") pod \"kube-auth-proxy-6dd88dbbb7-48gc7\" (UID: \"20362046-2f57-4bea-8b2d-4a9092f60059\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"
Nov 03 19:46:19.646598 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.646423    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20362046-2f57-4bea-8b2d-4a9092f60059-tls-certs\") pod \"kube-auth-proxy-6dd88dbbb7-48gc7\" (UID: \"20362046-2f57-4bea-8b2d-4a9092f60059\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"
Nov 03 19:46:19.646598 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.646500    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20362046-2f57-4bea-8b2d-4a9092f60059-tmp\") pod \"kube-auth-proxy-6dd88dbbb7-48gc7\" (UID: \"20362046-2f57-4bea-8b2d-4a9092f60059\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"
Nov 03 19:46:19.648689 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.648668    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20362046-2f57-4bea-8b2d-4a9092f60059-tmp\") pod \"kube-auth-proxy-6dd88dbbb7-48gc7\" (UID: \"20362046-2f57-4bea-8b2d-4a9092f60059\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"
Nov 03 19:46:19.648890 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.648872    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20362046-2f57-4bea-8b2d-4a9092f60059-tls-certs\") pod \"kube-auth-proxy-6dd88dbbb7-48gc7\" (UID: \"20362046-2f57-4bea-8b2d-4a9092f60059\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"
Nov 03 19:46:19.657531 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.657504    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6w4g\" (UniqueName: \"kubernetes.io/projected/20362046-2f57-4bea-8b2d-4a9092f60059-kube-api-access-c6w4g\") pod \"kube-auth-proxy-6dd88dbbb7-48gc7\" (UID: \"20362046-2f57-4bea-8b2d-4a9092f60059\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"
Nov 03 19:46:19.741340 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:19.741205    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"
Nov 03 19:46:20.042137 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:20.042110    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7"]
Nov 03 19:46:20.044299 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:46:20.044275    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20362046_2f57_4bea_8b2d_4a9092f60059.slice/crio-5899c14e852db96f8093e3f3177f12d5c27b8c9d0179e97d8036a77fb64c7c82 WatchSource:0}: Error finding container 5899c14e852db96f8093e3f3177f12d5c27b8c9d0179e97d8036a77fb64c7c82: Status 404 returned error can't find the container with id 5899c14e852db96f8093e3f3177f12d5c27b8c9d0179e97d8036a77fb64c7c82
Nov 03 19:46:20.229612 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:20.229578    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7" event={"ID":"20362046-2f57-4bea-8b2d-4a9092f60059","Type":"ContainerStarted","Data":"5899c14e852db96f8093e3f3177f12d5c27b8c9d0179e97d8036a77fb64c7c82"}
Nov 03 19:46:21.639587 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.639554    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"]
Nov 03 19:46:21.648373 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.648350    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.658209 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.658137    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"]
Nov 03 19:46:21.672903 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.672871    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-hl2dr\""
Nov 03 19:46:21.686186 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.686066    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 19:46:21.686186 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.686161    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 19:46:21.689799 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.689777    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 19:46:21.766568 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.766535    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp578\" (UniqueName: \"kubernetes.io/projected/7121959d-7966-4626-addd-371fa2295cae-kube-api-access-xp578\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.766743 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.766690    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7121959d-7966-4626-addd-371fa2295cae-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.766743 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.766730    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7121959d-7966-4626-addd-371fa2295cae-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.766862 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.766755    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7121959d-7966-4626-addd-371fa2295cae-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.867674 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.867631    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp578\" (UniqueName: \"kubernetes.io/projected/7121959d-7966-4626-addd-371fa2295cae-kube-api-access-xp578\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.867846 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.867742    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7121959d-7966-4626-addd-371fa2295cae-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.867846 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.867774    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7121959d-7966-4626-addd-371fa2295cae-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.867846 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.867798    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7121959d-7966-4626-addd-371fa2295cae-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.868169 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.868151    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7121959d-7966-4626-addd-371fa2295cae-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.869219 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.869199    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7121959d-7966-4626-addd-371fa2295cae-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.869478 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.869458    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7121959d-7966-4626-addd-371fa2295cae-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.876657 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.876631    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp578\" (UniqueName: \"kubernetes.io/projected/7121959d-7966-4626-addd-371fa2295cae-kube-api-access-xp578\") pod \"kserve-module-controller-manager-7566cf5fc8-g487l\" (UID: \"7121959d-7966-4626-addd-371fa2295cae\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:21.966629 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:21.966558    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:22.251157 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:22.251072    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"]
Nov 03 19:46:22.253621 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:46:22.253593    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7121959d_7966_4626_addd_371fa2295cae.slice/crio-d53ca2bc4022293cfc2a5bfa3a93dc8570d8e0757a94b7b7d013ebe9f88f2995 WatchSource:0}: Error finding container d53ca2bc4022293cfc2a5bfa3a93dc8570d8e0757a94b7b7d013ebe9f88f2995: Status 404 returned error can't find the container with id d53ca2bc4022293cfc2a5bfa3a93dc8570d8e0757a94b7b7d013ebe9f88f2995
Nov 03 19:46:23.245833 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:23.245796    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7" event={"ID":"20362046-2f57-4bea-8b2d-4a9092f60059","Type":"ContainerStarted","Data":"c9e70ba1947abfe9badb678542d2167553a6d1d8689c994e42d608af0e259255"}
Nov 03 19:46:23.246951 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:23.246929    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l" event={"ID":"7121959d-7966-4626-addd-371fa2295cae","Type":"ContainerStarted","Data":"d53ca2bc4022293cfc2a5bfa3a93dc8570d8e0757a94b7b7d013ebe9f88f2995"}
Nov 03 19:46:23.262220 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:23.262164    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-48gc7" podStartSLOduration=1.903157941 podStartE2EDuration="4.262146775s" totalImagesPullingTime="2.358988834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:23.260470783 +0000 UTC m=+365.773292564" watchObservedRunningTime="2026-11-03 19:46:23.262146775 +0000 UTC m=+365.774968559"
Nov 03 19:46:28.271539 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:28.271507    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l" event={"ID":"7121959d-7966-4626-addd-371fa2295cae","Type":"ContainerStarted","Data":"1d31a87a7acc2b9c1b0423cc44e11fe7d60227691f1b5aa91944dabb2d7c26d3"}
Nov 03 19:46:28.271916 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:28.271609    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:28.290809 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:28.290758    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l" podStartSLOduration=1.403641642 podStartE2EDuration="7.290742602s" totalImagesPullingTime="5.88710096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:28.289291956 +0000 UTC m=+370.802113774" watchObservedRunningTime="2026-11-03 19:46:28.290742602 +0000 UTC m=+370.803564554"
Nov 03 19:46:28.796236 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:46:28.796203    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:46:29.226549 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:29.226521    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:46:49.279193 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:46:49.279163    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-g487l"
Nov 03 19:46:58.802826 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:46:58.802785    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:47:17.800346 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.800318    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"]
Nov 03 19:47:17.808661 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.808630    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:47:17.814019 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.813990    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"]
Nov 03 19:47:17.814272 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.814258    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 19:47:17.814362 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.814343    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-r2txh\""
Nov 03 19:47:17.843701 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.843663    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ade14a9-328f-4aa0-ad17-3268d284d25e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5g8ml\" (UID: \"5ade14a9-328f-4aa0-ad17-3268d284d25e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:47:17.843866 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.843709    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q52g9\" (UniqueName: \"kubernetes.io/projected/5ade14a9-328f-4aa0-ad17-3268d284d25e-kube-api-access-q52g9\") pod \"llmisvc-controller-manager-869d5dcc5d-5g8ml\" (UID: \"5ade14a9-328f-4aa0-ad17-3268d284d25e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:47:17.854715 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.854689    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-zt4t7"]
Nov 03 19:47:17.858375 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.858356    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:17.869995 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.869972    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zt4t7"]
Nov 03 19:47:17.914908 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.914867    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 19:47:17.914908 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.914880    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 19:47:17.915164 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.914929    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-v54sc\""
Nov 03 19:47:17.945146 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.945114    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lnd\" (UniqueName: \"kubernetes.io/projected/eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8-kube-api-access-b5lnd\") pod \"odh-model-controller-769875db56-zt4t7\" (UID: \"eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8\") " pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:17.945327 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.945153    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ade14a9-328f-4aa0-ad17-3268d284d25e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5g8ml\" (UID: \"5ade14a9-328f-4aa0-ad17-3268d284d25e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:47:17.945327 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.945174    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q52g9\" (UniqueName: \"kubernetes.io/projected/5ade14a9-328f-4aa0-ad17-3268d284d25e-kube-api-access-q52g9\") pod \"llmisvc-controller-manager-869d5dcc5d-5g8ml\" (UID: \"5ade14a9-328f-4aa0-ad17-3268d284d25e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:47:17.945327 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.945245    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8-cert\") pod \"odh-model-controller-769875db56-zt4t7\" (UID: \"eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8\") " pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:17.945327 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.945294    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8-metrics-certs\") pod \"odh-model-controller-769875db56-zt4t7\" (UID: \"eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8\") " pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:17.947792 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.947768    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ade14a9-328f-4aa0-ad17-3268d284d25e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5g8ml\" (UID: \"5ade14a9-328f-4aa0-ad17-3268d284d25e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:47:17.953589 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:17.953563    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q52g9\" (UniqueName: \"kubernetes.io/projected/5ade14a9-328f-4aa0-ad17-3268d284d25e-kube-api-access-q52g9\") pod \"llmisvc-controller-manager-869d5dcc5d-5g8ml\" (UID: \"5ade14a9-328f-4aa0-ad17-3268d284d25e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:47:18.046105 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.046070    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8-cert\") pod \"odh-model-controller-769875db56-zt4t7\" (UID: \"eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8\") " pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:18.046275 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.046116    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8-metrics-certs\") pod \"odh-model-controller-769875db56-zt4t7\" (UID: \"eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8\") " pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:18.046275 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.046166    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lnd\" (UniqueName: \"kubernetes.io/projected/eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8-kube-api-access-b5lnd\") pod \"odh-model-controller-769875db56-zt4t7\" (UID: \"eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8\") " pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:18.048913 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.048881    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8-cert\") pod \"odh-model-controller-769875db56-zt4t7\" (UID: \"eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8\") " pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:18.048913 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.048907    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8-metrics-certs\") pod \"odh-model-controller-769875db56-zt4t7\" (UID: \"eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8\") " pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:18.053737 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.053684    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lnd\" (UniqueName: \"kubernetes.io/projected/eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8-kube-api-access-b5lnd\") pod \"odh-model-controller-769875db56-zt4t7\" (UID: \"eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8\") " pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:18.123037 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.123000    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:47:18.174087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.173792    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:18.408561 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.408538    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"]
Nov 03 19:47:18.410256 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:47:18.410220    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5ade14a9_328f_4aa0_ad17_3268d284d25e.slice/crio-fc2ac3fb13182adc8e1d75ed3ca7ff20b4e7094d2fdb6d70135924ceb6372a8a WatchSource:0}: Error finding container fc2ac3fb13182adc8e1d75ed3ca7ff20b4e7094d2fdb6d70135924ceb6372a8a: Status 404 returned error can't find the container with id fc2ac3fb13182adc8e1d75ed3ca7ff20b4e7094d2fdb6d70135924ceb6372a8a
Nov 03 19:47:18.478032 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.478007    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zt4t7"]
Nov 03 19:47:18.480548 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:47:18.480520    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4ab8ab_5b3b_4e1f_89e0_60fbaad953a8.slice/crio-6e16e3c9c9d588bc20e789f68c3f70a12c6eb124eb09e73cf2f378f576c8eb57 WatchSource:0}: Error finding container 6e16e3c9c9d588bc20e789f68c3f70a12c6eb124eb09e73cf2f378f576c8eb57: Status 404 returned error can't find the container with id 6e16e3c9c9d588bc20e789f68c3f70a12c6eb124eb09e73cf2f378f576c8eb57
Nov 03 19:47:18.508574 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.508539    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zt4t7" event={"ID":"eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8","Type":"ContainerStarted","Data":"6e16e3c9c9d588bc20e789f68c3f70a12c6eb124eb09e73cf2f378f576c8eb57"}
Nov 03 19:47:18.509746 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:18.509716    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml" event={"ID":"5ade14a9-328f-4aa0-ad17-3268d284d25e","Type":"ContainerStarted","Data":"fc2ac3fb13182adc8e1d75ed3ca7ff20b4e7094d2fdb6d70135924ceb6372a8a"}
Nov 03 19:47:21.529538 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:21.529498    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml" event={"ID":"5ade14a9-328f-4aa0-ad17-3268d284d25e","Type":"ContainerStarted","Data":"93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef"}
Nov 03 19:47:21.530037 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:21.529566    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:47:21.546073 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:21.546027    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml" podStartSLOduration=2.226924611 podStartE2EDuration="4.546014427s" totalImagesPullingTime="2.319089816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:21.545919721 +0000 UTC m=+424.058741504" watchObservedRunningTime="2026-11-03 19:47:21.546014427 +0000 UTC m=+424.058836208"
Nov 03 19:47:23.541883 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:23.541845    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zt4t7" event={"ID":"eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8","Type":"ContainerStarted","Data":"1dcda08b5a66dcd1ca50317d167c4133caa2b389493ca068a6867064beb12ec0"}
Nov 03 19:47:23.542264 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:23.541958    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:23.561288 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:23.561236    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-zt4t7" podStartSLOduration=2.170165406 podStartE2EDuration="6.561223117s" totalImagesPullingTime="4.391057711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:23.559910913 +0000 UTC m=+426.072732697" watchObservedRunningTime="2026-11-03 19:47:23.561223117 +0000 UTC m=+426.074044896"
Nov 03 19:47:28.793717 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:47:28.793676    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:47:34.550510 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:34.550481    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-zt4t7"
Nov 03 19:47:52.538536 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:47:52.538502    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:47:58.798582 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:47:58.798494    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:48:28.794499 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:48:28.794459    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:48:58.793598 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:48:58.793564    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:49:28.794737 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:49:28.794632    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:49:58.794099 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:49:58.794063    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:50:28.793941 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:50:28.793898    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:50:58.794459 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:50:58.794350    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:51:28.793992 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:51:28.793957    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:51:58.794841 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:51:58.794799    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:52:28.794662 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:52:28.794578    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:52:58.793074 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:52:58.793032    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:53:28.794290 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:53:28.794247    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:53:58.794253 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:53:58.794172    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:54:28.792766 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:54:28.792724    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:54:58.793861 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:54:58.793824    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:55:28.794853 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:55:28.794773    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:55:58.793538 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:55:58.793495    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:56:28.793994 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:56:28.793955    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:56:38.718218 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:38.718184    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zwbxr_b4a89640-183b-456f-8f5a-5307275dd08c/manager/0.log"
Nov 03 19:56:38.914961 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:38.914928    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"]
Nov 03 19:56:38.915141 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:38.915120    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr" podUID="b4a89640-183b-456f-8f5a-5307275dd08c" containerName="manager" containerID="cri-o://56932ee357a049c61aca0e508d9d797ed6d80e0524ea009c21336a3ad8721564" gracePeriod=10
Nov 03 19:56:38.939926 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:38.939900    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"]
Nov 03 19:56:38.943500 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:38.943479    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 19:56:38.955869 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:38.955844    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"]
Nov 03 19:56:39.046653 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.046616    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85077442-78c8-48dc-85f1-ce4bb58bd59c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7v42t\" (UID: \"85077442-78c8-48dc-85f1-ce4bb58bd59c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 19:56:39.046778 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.046657    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwsw7\" (UniqueName: \"kubernetes.io/projected/85077442-78c8-48dc-85f1-ce4bb58bd59c-kube-api-access-rwsw7\") pod \"kuadrant-operator-controller-manager-678d87766f-7v42t\" (UID: \"85077442-78c8-48dc-85f1-ce4bb58bd59c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 19:56:39.131580 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.131552    5600 generic.go:356] "Generic (PLEG): container finished" podID="b4a89640-183b-456f-8f5a-5307275dd08c" containerID="56932ee357a049c61aca0e508d9d797ed6d80e0524ea009c21336a3ad8721564" exitCode=0
Nov 03 19:56:39.131751 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.131627    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr" event={"ID":"b4a89640-183b-456f-8f5a-5307275dd08c","Type":"ContainerDied","Data":"56932ee357a049c61aca0e508d9d797ed6d80e0524ea009c21336a3ad8721564"}
Nov 03 19:56:39.148048 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.148021    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85077442-78c8-48dc-85f1-ce4bb58bd59c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7v42t\" (UID: \"85077442-78c8-48dc-85f1-ce4bb58bd59c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 19:56:39.148220 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.148051    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwsw7\" (UniqueName: \"kubernetes.io/projected/85077442-78c8-48dc-85f1-ce4bb58bd59c-kube-api-access-rwsw7\") pod \"kuadrant-operator-controller-manager-678d87766f-7v42t\" (UID: \"85077442-78c8-48dc-85f1-ce4bb58bd59c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 19:56:39.148448 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.148417    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85077442-78c8-48dc-85f1-ce4bb58bd59c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7v42t\" (UID: \"85077442-78c8-48dc-85f1-ce4bb58bd59c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 19:56:39.158887 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.158859    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwsw7\" (UniqueName: \"kubernetes.io/projected/85077442-78c8-48dc-85f1-ce4bb58bd59c-kube-api-access-rwsw7\") pod \"kuadrant-operator-controller-manager-678d87766f-7v42t\" (UID: \"85077442-78c8-48dc-85f1-ce4bb58bd59c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 19:56:39.173975 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.173956    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:56:39.249261 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.249179    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a89640-183b-456f-8f5a-5307275dd08c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a89640-183b-456f-8f5a-5307275dd08c-extensions-socket-volume\") pod \"b4a89640-183b-456f-8f5a-5307275dd08c\" (UID: \"b4a89640-183b-456f-8f5a-5307275dd08c\") "
Nov 03 19:56:39.249261 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.249251    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a89640-183b-456f-8f5a-5307275dd08c-kube-api-access-r6m94\" (UniqueName: \"kubernetes.io/projected/b4a89640-183b-456f-8f5a-5307275dd08c-kube-api-access-r6m94\") pod \"b4a89640-183b-456f-8f5a-5307275dd08c\" (UID: \"b4a89640-183b-456f-8f5a-5307275dd08c\") "
Nov 03 19:56:39.249609 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.249586    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a89640-183b-456f-8f5a-5307275dd08c-extensions-socket-volume" pod "b4a89640-183b-456f-8f5a-5307275dd08c" (UID: "b4a89640-183b-456f-8f5a-5307275dd08c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:56:39.251534 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.251500    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a89640-183b-456f-8f5a-5307275dd08c-kube-api-access-r6m94" pod "b4a89640-183b-456f-8f5a-5307275dd08c" (UID: "b4a89640-183b-456f-8f5a-5307275dd08c"). InnerVolumeSpecName "kube-api-access-r6m94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:39.300923 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.300892    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 19:56:39.350597 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.350567    5600 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a89640-183b-456f-8f5a-5307275dd08c-extensions-socket-volume\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:39.350728 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.350602    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6m94\" (UniqueName: \"kubernetes.io/projected/b4a89640-183b-456f-8f5a-5307275dd08c-kube-api-access-r6m94\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:39.586958 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:39.586935    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"]
Nov 03 19:56:39.589308 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:56:39.589279    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85077442_78c8_48dc_85f1_ce4bb58bd59c.slice/crio-afbef16fb943ff7c51d188f5c8cdee4334306b159f55f3ff6bd4aa56c285217d WatchSource:0}: Error finding container afbef16fb943ff7c51d188f5c8cdee4334306b159f55f3ff6bd4aa56c285217d: Status 404 returned error can't find the container with id afbef16fb943ff7c51d188f5c8cdee4334306b159f55f3ff6bd4aa56c285217d
Nov 03 19:56:40.138808 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:40.138775    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t" event={"ID":"85077442-78c8-48dc-85f1-ce4bb58bd59c","Type":"ContainerStarted","Data":"d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b"}
Nov 03 19:56:40.139292 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:40.138825    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t" event={"ID":"85077442-78c8-48dc-85f1-ce4bb58bd59c","Type":"ContainerStarted","Data":"afbef16fb943ff7c51d188f5c8cdee4334306b159f55f3ff6bd4aa56c285217d"}
Nov 03 19:56:40.139292 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:40.138855    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 19:56:40.140160 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:40.140135    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr" event={"ID":"b4a89640-183b-456f-8f5a-5307275dd08c","Type":"ContainerDied","Data":"531f5ea0da8b15089c4e3fd92b324f630b0049c1897376efbe5cf96328be14d3"}
Nov 03 19:56:40.140240 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:40.140183    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"
Nov 03 19:56:40.140277 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:40.140190    5600 scope.go:122] "RemoveContainer" containerID="56932ee357a049c61aca0e508d9d797ed6d80e0524ea009c21336a3ad8721564"
Nov 03 19:56:40.163086 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:40.163043    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t" podStartSLOduration=2.163028458 podStartE2EDuration="2.163028458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:40.16203362 +0000 UTC m=+982.674855401" watchObservedRunningTime="2026-11-03 19:56:40.163028458 +0000 UTC m=+982.675850241"
Nov 03 19:56:40.181907 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:40.181883    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"]
Nov 03 19:56:40.185724 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:40.185702    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zwbxr"]
Nov 03 19:56:41.614614 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:41.614579    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a89640-183b-456f-8f5a-5307275dd08c" path="/var/lib/kubelet/pods/b4a89640-183b-456f-8f5a-5307275dd08c/volumes"
Nov 03 19:56:51.148846 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:51.148818    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 19:56:56.953075 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:56.952993    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rnv65"]
Nov 03 19:56:56.953463 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:56.953344    5600 cpu_manager.go:414] "containerMap: removing container" podUID="b4a89640-183b-456f-8f5a-5307275dd08c" containerName="manager"
Nov 03 19:56:56.953463 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:56.953354    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a89640-183b-456f-8f5a-5307275dd08c" containerName="manager"
Nov 03 19:56:56.953463 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:56.953426    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a89640-183b-456f-8f5a-5307275dd08c" containerName="manager"
Nov 03 19:56:56.957805 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:56.957781    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:56:56.964148 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:56.964117    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rnv65"]
Nov 03 19:56:56.965966 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:56.964872    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 19:56:56.965966 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:56.965045    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lpjdc\""
Nov 03 19:56:57.058559 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.058526    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rnv65"]
Nov 03 19:56:57.122658 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.122623    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7258e5a1-231c-4bc9-a335-38799cef9551-config-file\") pod \"limitador-limitador-74dfcfb894-rnv65\" (UID: \"7258e5a1-231c-4bc9-a335-38799cef9551\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:56:57.122658 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.122663    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzml\" (UniqueName: \"kubernetes.io/projected/7258e5a1-231c-4bc9-a335-38799cef9551-kube-api-access-tvzml\") pod \"limitador-limitador-74dfcfb894-rnv65\" (UID: \"7258e5a1-231c-4bc9-a335-38799cef9551\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:56:57.223301 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.223213    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7258e5a1-231c-4bc9-a335-38799cef9551-config-file\") pod \"limitador-limitador-74dfcfb894-rnv65\" (UID: \"7258e5a1-231c-4bc9-a335-38799cef9551\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:56:57.223301 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.223250    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzml\" (UniqueName: \"kubernetes.io/projected/7258e5a1-231c-4bc9-a335-38799cef9551-kube-api-access-tvzml\") pod \"limitador-limitador-74dfcfb894-rnv65\" (UID: \"7258e5a1-231c-4bc9-a335-38799cef9551\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:56:57.224084 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.224058    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7258e5a1-231c-4bc9-a335-38799cef9551-config-file\") pod \"limitador-limitador-74dfcfb894-rnv65\" (UID: \"7258e5a1-231c-4bc9-a335-38799cef9551\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:56:57.231999 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.231966    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzml\" (UniqueName: \"kubernetes.io/projected/7258e5a1-231c-4bc9-a335-38799cef9551-kube-api-access-tvzml\") pod \"limitador-limitador-74dfcfb894-rnv65\" (UID: \"7258e5a1-231c-4bc9-a335-38799cef9551\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:56:57.272809 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.272778    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:56:57.553213 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.553187    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rnv65"]
Nov 03 19:56:57.555547 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:56:57.555517    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7258e5a1_231c_4bc9_a335_38799cef9551.slice/crio-8a4470ec811f1b803990053523d6342fc91c7d96dfd567b8611c490b55a32041 WatchSource:0}: Error finding container 8a4470ec811f1b803990053523d6342fc91c7d96dfd567b8611c490b55a32041: Status 404 returned error can't find the container with id 8a4470ec811f1b803990053523d6342fc91c7d96dfd567b8611c490b55a32041
Nov 03 19:56:57.557764 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.557745    5600 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:56:57.907264 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.907230    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8t2jh"]
Nov 03 19:56:57.911884 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.911863    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh"
Nov 03 19:56:57.914764 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.914738    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8t2jh"]
Nov 03 19:56:57.918235 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:57.918201    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n72st\""
Nov 03 19:56:58.029544 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:58.029499    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47hf\" (UniqueName: \"kubernetes.io/projected/ba7f90cf-6e12-49ce-a541-95d5679edf9c-kube-api-access-w47hf\") pod \"authorino-56c6c9ddd5-8t2jh\" (UID: \"ba7f90cf-6e12-49ce-a541-95d5679edf9c\") " pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh"
Nov 03 19:56:58.129948 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:58.129914    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47hf\" (UniqueName: \"kubernetes.io/projected/ba7f90cf-6e12-49ce-a541-95d5679edf9c-kube-api-access-w47hf\") pod \"authorino-56c6c9ddd5-8t2jh\" (UID: \"ba7f90cf-6e12-49ce-a541-95d5679edf9c\") " pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh"
Nov 03 19:56:58.138135 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:58.138102    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47hf\" (UniqueName: \"kubernetes.io/projected/ba7f90cf-6e12-49ce-a541-95d5679edf9c-kube-api-access-w47hf\") pod \"authorino-56c6c9ddd5-8t2jh\" (UID: \"ba7f90cf-6e12-49ce-a541-95d5679edf9c\") " pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh"
Nov 03 19:56:58.226832 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:58.226752    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh"
Nov 03 19:56:58.232388 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:58.232364    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65" event={"ID":"7258e5a1-231c-4bc9-a335-38799cef9551","Type":"ContainerStarted","Data":"8a4470ec811f1b803990053523d6342fc91c7d96dfd567b8611c490b55a32041"}
Nov 03 19:56:58.499858 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:58.499824    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8t2jh"]
Nov 03 19:56:58.501829 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:56:58.501799    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba7f90cf_6e12_49ce_a541_95d5679edf9c.slice/crio-caf44e0a98534f0a930b670e7654694724363bbfb313cf40c125a0bcf53f9c5a WatchSource:0}: Error finding container caf44e0a98534f0a930b670e7654694724363bbfb313cf40c125a0bcf53f9c5a: Status 404 returned error can't find the container with id caf44e0a98534f0a930b670e7654694724363bbfb313cf40c125a0bcf53f9c5a
Nov 03 19:56:58.800317 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:56:58.800232    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:56:59.238983 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:59.238944    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65" event={"ID":"7258e5a1-231c-4bc9-a335-38799cef9551","Type":"ContainerStarted","Data":"acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0"}
Nov 03 19:56:59.239516 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:59.239079    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:56:59.240223 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:59.240200    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh" event={"ID":"ba7f90cf-6e12-49ce-a541-95d5679edf9c","Type":"ContainerStarted","Data":"caf44e0a98534f0a930b670e7654694724363bbfb313cf40c125a0bcf53f9c5a"}
Nov 03 19:56:59.258038 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:56:59.257989    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65" podStartSLOduration=1.933665417 podStartE2EDuration="3.257975205s" totalImagesPullingTime="1.324309788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:59.256636431 +0000 UTC m=+1001.769458246" watchObservedRunningTime="2026-11-03 19:56:59.257975205 +0000 UTC m=+1001.770796986"
Nov 03 19:57:01.250675 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:01.250637    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh" event={"ID":"ba7f90cf-6e12-49ce-a541-95d5679edf9c","Type":"ContainerStarted","Data":"8202be8683c0fce0141841d09aaa6bd20313d2c0498150c9704ff724560ece40"}
Nov 03 19:57:01.266930 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:01.266877    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh" podStartSLOduration=2.066480712 podStartE2EDuration="4.266860871s" totalImagesPullingTime="2.200380159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:01.265008585 +0000 UTC m=+1003.777830368" watchObservedRunningTime="2026-11-03 19:57:01.266860871 +0000 UTC m=+1003.779682653"
Nov 03 19:57:10.246707 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:10.246675    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:57:12.356728 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:12.356691    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rnv65"]
Nov 03 19:57:12.357128 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:12.356897    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65" podUID="7258e5a1-231c-4bc9-a335-38799cef9551" containerName="limitador" containerID="cri-o://acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0" gracePeriod=30
Nov 03 19:57:12.897744 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:12.897721    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:57:12.961268 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:12.961185    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7258e5a1-231c-4bc9-a335-38799cef9551-kube-api-access-tvzml\" (UniqueName: \"kubernetes.io/projected/7258e5a1-231c-4bc9-a335-38799cef9551-kube-api-access-tvzml\") pod \"7258e5a1-231c-4bc9-a335-38799cef9551\" (UID: \"7258e5a1-231c-4bc9-a335-38799cef9551\") "
Nov 03 19:57:12.961268 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:12.961232    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7258e5a1-231c-4bc9-a335-38799cef9551-config-file\" (UniqueName: \"kubernetes.io/configmap/7258e5a1-231c-4bc9-a335-38799cef9551-config-file\") pod \"7258e5a1-231c-4bc9-a335-38799cef9551\" (UID: \"7258e5a1-231c-4bc9-a335-38799cef9551\") "
Nov 03 19:57:12.961675 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:12.961641    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7258e5a1-231c-4bc9-a335-38799cef9551-config-file" pod "7258e5a1-231c-4bc9-a335-38799cef9551" (UID: "7258e5a1-231c-4bc9-a335-38799cef9551"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:12.963451 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:12.963417    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7258e5a1-231c-4bc9-a335-38799cef9551-kube-api-access-tvzml" pod "7258e5a1-231c-4bc9-a335-38799cef9551" (UID: "7258e5a1-231c-4bc9-a335-38799cef9551"). InnerVolumeSpecName "kube-api-access-tvzml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:13.062471 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.062414    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvzml\" (UniqueName: \"kubernetes.io/projected/7258e5a1-231c-4bc9-a335-38799cef9551-kube-api-access-tvzml\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:13.062471 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.062466    5600 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7258e5a1-231c-4bc9-a335-38799cef9551-config-file\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:13.309565 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.309470    5600 generic.go:356] "Generic (PLEG): container finished" podID="7258e5a1-231c-4bc9-a335-38799cef9551" containerID="acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0" exitCode=0
Nov 03 19:57:13.309565 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.309509    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65"
Nov 03 19:57:13.309565 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.309550    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65" event={"ID":"7258e5a1-231c-4bc9-a335-38799cef9551","Type":"ContainerDied","Data":"acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0"}
Nov 03 19:57:13.309783 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.309601    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-rnv65" event={"ID":"7258e5a1-231c-4bc9-a335-38799cef9551","Type":"ContainerDied","Data":"8a4470ec811f1b803990053523d6342fc91c7d96dfd567b8611c490b55a32041"}
Nov 03 19:57:13.309783 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.309623    5600 scope.go:122] "RemoveContainer" containerID="acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0"
Nov 03 19:57:13.319087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.319072    5600 scope.go:122] "RemoveContainer" containerID="acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0"
Nov 03 19:57:13.319348 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:57:13.319328    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0\": container with ID starting with acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0 not found: ID does not exist" containerID="acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0"
Nov 03 19:57:13.319391 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.319356    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0"} err="failed to get container status \"acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0\": rpc error: code = NotFound desc = could not find container \"acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0\": container with ID starting with acca270b5907c09e67053104f0bc7dd9ccb89a8a5a1267799b3b233e6cb6d0b0 not found: ID does not exist"
Nov 03 19:57:13.332142 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.332119    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rnv65"]
Nov 03 19:57:13.338236 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.338217    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rnv65"]
Nov 03 19:57:13.614314 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:13.614241    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7258e5a1-231c-4bc9-a335-38799cef9551" path="/var/lib/kubelet/pods/7258e5a1-231c-4bc9-a335-38799cef9551/volumes"
Nov 03 19:57:18.764346 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.764306    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"]
Nov 03 19:57:18.764756 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.764690    5600 cpu_manager.go:414] "containerMap: removing container" podUID="7258e5a1-231c-4bc9-a335-38799cef9551" containerName="limitador"
Nov 03 19:57:18.764756 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.764702    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7258e5a1-231c-4bc9-a335-38799cef9551" containerName="limitador"
Nov 03 19:57:18.764756 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.764757    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="7258e5a1-231c-4bc9-a335-38799cef9551" containerName="limitador"
Nov 03 19:57:18.771234 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.771213    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"
Nov 03 19:57:18.776640 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.776614    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:57:18.776782 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.776749    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"]
Nov 03 19:57:18.777594 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.777572    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 19:57:18.777594 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.777581    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qk8br\""
Nov 03 19:57:18.777733 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.777604    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:57:18.812963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.812937    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/51d053ff-ea16-4ffd-ac26-ad845dce0b82-data\") pod \"postgres-7cdbf6dc44-q9b6f\" (UID: \"51d053ff-ea16-4ffd-ac26-ad845dce0b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"
Nov 03 19:57:18.813125 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.812983    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jxn\" (UniqueName: \"kubernetes.io/projected/51d053ff-ea16-4ffd-ac26-ad845dce0b82-kube-api-access-g2jxn\") pod \"postgres-7cdbf6dc44-q9b6f\" (UID: \"51d053ff-ea16-4ffd-ac26-ad845dce0b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"
Nov 03 19:57:18.914577 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.914538    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/51d053ff-ea16-4ffd-ac26-ad845dce0b82-data\") pod \"postgres-7cdbf6dc44-q9b6f\" (UID: \"51d053ff-ea16-4ffd-ac26-ad845dce0b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"
Nov 03 19:57:18.914720 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.914612    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jxn\" (UniqueName: \"kubernetes.io/projected/51d053ff-ea16-4ffd-ac26-ad845dce0b82-kube-api-access-g2jxn\") pod \"postgres-7cdbf6dc44-q9b6f\" (UID: \"51d053ff-ea16-4ffd-ac26-ad845dce0b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"
Nov 03 19:57:18.914932 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.914915    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/51d053ff-ea16-4ffd-ac26-ad845dce0b82-data\") pod \"postgres-7cdbf6dc44-q9b6f\" (UID: \"51d053ff-ea16-4ffd-ac26-ad845dce0b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"
Nov 03 19:57:18.926556 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:18.926522    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jxn\" (UniqueName: \"kubernetes.io/projected/51d053ff-ea16-4ffd-ac26-ad845dce0b82-kube-api-access-g2jxn\") pod \"postgres-7cdbf6dc44-q9b6f\" (UID: \"51d053ff-ea16-4ffd-ac26-ad845dce0b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"
Nov 03 19:57:19.085603 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:19.085517    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"
Nov 03 19:57:19.367190 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:19.367168    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"]
Nov 03 19:57:19.369155 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:57:19.369121    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d053ff_ea16_4ffd_ac26_ad845dce0b82.slice/crio-ceae2e44f6fa78b48dcf21a6c4b0d9b89a6a88961a51dfbdff92291521f66b9f WatchSource:0}: Error finding container ceae2e44f6fa78b48dcf21a6c4b0d9b89a6a88961a51dfbdff92291521f66b9f: Status 404 returned error can't find the container with id ceae2e44f6fa78b48dcf21a6c4b0d9b89a6a88961a51dfbdff92291521f66b9f
Nov 03 19:57:20.349708 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:20.349672    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f" event={"ID":"51d053ff-ea16-4ffd-ac26-ad845dce0b82","Type":"ContainerStarted","Data":"ceae2e44f6fa78b48dcf21a6c4b0d9b89a6a88961a51dfbdff92291521f66b9f"}
Nov 03 19:57:25.383506 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:25.383461    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f" event={"ID":"51d053ff-ea16-4ffd-ac26-ad845dce0b82","Type":"ContainerStarted","Data":"8bdcab6a011656f89cb2562d213db11bb64bb2398ad096fbbca82b4409c71f37"}
Nov 03 19:57:25.383891 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:25.383604    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"
Nov 03 19:57:25.400666 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:25.400603    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f" podStartSLOduration=1.5393026779999999 podStartE2EDuration="7.40058443s" totalImagesPullingTime="5.861281752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:25.398383925 +0000 UTC m=+1027.911205708" watchObservedRunningTime="2026-11-03 19:57:25.40058443 +0000 UTC m=+1027.913406211"
Nov 03 19:57:28.794480 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:57:28.794427    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:57:31.418900 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:31.418873    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-q9b6f"
Nov 03 19:57:43.777920 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:43.777874    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"]
Nov 03 19:57:43.782898 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:43.782873    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:43.791212 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:43.791186    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-pn4bk\""
Nov 03 19:57:43.792954 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:43.792935    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 19:57:43.793096 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:43.792955    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 19:57:43.793197 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:43.793040    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 19:57:43.796077 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:43.796047    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"]
Nov 03 19:57:43.943013 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:43.942979    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp9jf\" (UniqueName: \"kubernetes.io/projected/05f7ff11-6561-4688-b7e8-5b518643fe5a-kube-api-access-tp9jf\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:43.943013 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:43.943015    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05f7ff11-6561-4688-b7e8-5b518643fe5a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:43.943262 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:43.943166    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/05f7ff11-6561-4688-b7e8-5b518643fe5a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:43.943302 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:43.943261    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05f7ff11-6561-4688-b7e8-5b518643fe5a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:44.044327 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:44.044243    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/05f7ff11-6561-4688-b7e8-5b518643fe5a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:44.044327 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:44.044296    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05f7ff11-6561-4688-b7e8-5b518643fe5a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:44.044327 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:44.044326    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9jf\" (UniqueName: \"kubernetes.io/projected/05f7ff11-6561-4688-b7e8-5b518643fe5a-kube-api-access-tp9jf\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:44.044589 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:44.044345    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05f7ff11-6561-4688-b7e8-5b518643fe5a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:44.044971 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:44.044948    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05f7ff11-6561-4688-b7e8-5b518643fe5a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:44.045224 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:44.045193    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/05f7ff11-6561-4688-b7e8-5b518643fe5a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:44.046933 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:44.046914    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05f7ff11-6561-4688-b7e8-5b518643fe5a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:44.052380 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:44.052359    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp9jf\" (UniqueName: \"kubernetes.io/projected/05f7ff11-6561-4688-b7e8-5b518643fe5a-kube-api-access-tp9jf\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-2qb6d\" (UID: \"05f7ff11-6561-4688-b7e8-5b518643fe5a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:44.100756 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:44.100723    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:44.372177 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:44.372153    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"]
Nov 03 19:57:44.374402 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:57:44.374375    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f7ff11_6561_4688_b7e8_5b518643fe5a.slice/crio-b1184eaed56c9206f57c04f267781f7f56777d6046bfa92a52cfb34c89327f90 WatchSource:0}: Error finding container b1184eaed56c9206f57c04f267781f7f56777d6046bfa92a52cfb34c89327f90: Status 404 returned error can't find the container with id b1184eaed56c9206f57c04f267781f7f56777d6046bfa92a52cfb34c89327f90
Nov 03 19:57:44.479637 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:44.479602    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d" event={"ID":"05f7ff11-6561-4688-b7e8-5b518643fe5a","Type":"ContainerStarted","Data":"b1184eaed56c9206f57c04f267781f7f56777d6046bfa92a52cfb34c89327f90"}
Nov 03 19:57:58.562369 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:58.562317    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d" event={"ID":"05f7ff11-6561-4688-b7e8-5b518643fe5a","Type":"ContainerStarted","Data":"3775eca1eef6ad034ba3e47cec1bcbf4dc3843c09b6e8e751f4483a1ac5a7958"}
Nov 03 19:57:58.562842 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:58.562452    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:57:58.579963 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:57:58.579918    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d" podStartSLOduration=1.885463651 podStartE2EDuration="15.579904633s" totalImagesPullingTime="13.694440982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:58.578183324 +0000 UTC m=+1061.091005107" watchObservedRunningTime="2026-11-03 19:57:58.579904633 +0000 UTC m=+1061.092726414"
Nov 03 19:57:58.793662 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:57:58.793619    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:58:09.573460 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:09.573403    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-2qb6d"
Nov 03 19:58:15.583594 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.583557    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"]
Nov 03 19:58:15.587281 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.587259    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.598283 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.598259    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 19:58:15.598462 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.598302    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7bzpm\""
Nov 03 19:58:15.598462 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.598381    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 19:58:15.606152 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.606120    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"]
Nov 03 19:58:15.722544 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.722455    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548f11a1-81b3-4b10-83a7-05e320e509b2-openshift-service-ca\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.722544 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.722498    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-maas-api-tls\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.722544 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.722518    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4nx\" (UniqueName: \"kubernetes.io/projected/548f11a1-81b3-4b10-83a7-05e320e509b2-kube-api-access-bn4nx\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.722752 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.722649    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-metrics-certs\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.823839 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.823804    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548f11a1-81b3-4b10-83a7-05e320e509b2-openshift-service-ca\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.823839 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.823845    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-maas-api-tls\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.824084 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.823862    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4nx\" (UniqueName: \"kubernetes.io/projected/548f11a1-81b3-4b10-83a7-05e320e509b2-kube-api-access-bn4nx\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.824084 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.823899    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-metrics-certs\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.824685 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.824618    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548f11a1-81b3-4b10-83a7-05e320e509b2-openshift-service-ca\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.826536 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.826519    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-metrics-certs\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.826610 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.826573    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-maas-api-tls\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.832112 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.832091    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4nx\" (UniqueName: \"kubernetes.io/projected/548f11a1-81b3-4b10-83a7-05e320e509b2-kube-api-access-bn4nx\") pod \"maas-api-6d669c7fcd-z8r9g\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:15.901424 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:15.901399    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:16.176965 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:16.176878    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"]
Nov 03 19:58:16.179273 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:58:16.179239    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548f11a1_81b3_4b10_83a7_05e320e509b2.slice/crio-621112903d1a1a4c051368078132775228253baa141a28db12c96c1f1aeba25d WatchSource:0}: Error finding container 621112903d1a1a4c051368078132775228253baa141a28db12c96c1f1aeba25d: Status 404 returned error can't find the container with id 621112903d1a1a4c051368078132775228253baa141a28db12c96c1f1aeba25d
Nov 03 19:58:16.654469 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:16.654406    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g" event={"ID":"548f11a1-81b3-4b10-83a7-05e320e509b2","Type":"ContainerStarted","Data":"621112903d1a1a4c051368078132775228253baa141a28db12c96c1f1aeba25d"}
Nov 03 19:58:18.665806 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:18.665778    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g" event={"ID":"548f11a1-81b3-4b10-83a7-05e320e509b2","Type":"ContainerStarted","Data":"c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf"}
Nov 03 19:58:18.666145 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:18.665891    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:18.684529 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:18.684478    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g" podStartSLOduration=1.263567637 podStartE2EDuration="3.684464121s" totalImagesPullingTime="2.420896484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:18.682348999 +0000 UTC m=+1081.195170760" watchObservedRunningTime="2026-11-03 19:58:18.684464121 +0000 UTC m=+1081.197285903"
Nov 03 19:58:24.676132 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:24.676103    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:28.479020 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:28.478930    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8t2jh"]
Nov 03 19:58:28.479503 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:28.479165    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh" podUID="ba7f90cf-6e12-49ce-a541-95d5679edf9c" containerName="authorino" containerID="cri-o://8202be8683c0fce0141841d09aaa6bd20313d2c0498150c9704ff724560ece40" gracePeriod=30
Nov 03 19:58:28.716962 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:28.716930    5600 generic.go:356] "Generic (PLEG): container finished" podID="ba7f90cf-6e12-49ce-a541-95d5679edf9c" containerID="8202be8683c0fce0141841d09aaa6bd20313d2c0498150c9704ff724560ece40" exitCode=0
Nov 03 19:58:28.717113 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:28.716990    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh" event={"ID":"ba7f90cf-6e12-49ce-a541-95d5679edf9c","Type":"ContainerDied","Data":"8202be8683c0fce0141841d09aaa6bd20313d2c0498150c9704ff724560ece40"}
Nov 03 19:58:28.795346 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:58:28.795265    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:58:29.301311 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.301280    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh"
Nov 03 19:58:29.336517 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.336477    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"]
Nov 03 19:58:29.337002 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.336981    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ba7f90cf-6e12-49ce-a541-95d5679edf9c" containerName="authorino"
Nov 03 19:58:29.337119 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.337004    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7f90cf-6e12-49ce-a541-95d5679edf9c" containerName="authorino"
Nov 03 19:58:29.337119 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.337090    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7f90cf-6e12-49ce-a541-95d5679edf9c" containerName="authorino"
Nov 03 19:58:29.346283 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.346252    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"
Nov 03 19:58:29.348308 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.348276    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"]
Nov 03 19:58:29.449783 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.449704    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7f90cf-6e12-49ce-a541-95d5679edf9c-kube-api-access-w47hf\" (UniqueName: \"kubernetes.io/projected/ba7f90cf-6e12-49ce-a541-95d5679edf9c-kube-api-access-w47hf\") pod \"ba7f90cf-6e12-49ce-a541-95d5679edf9c\" (UID: \"ba7f90cf-6e12-49ce-a541-95d5679edf9c\") "
Nov 03 19:58:29.450079 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.449870    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bff4q\" (UniqueName: \"kubernetes.io/projected/ce3fa57a-f470-4d07-9cfc-1a3ece3df591-kube-api-access-bff4q\") pod \"llmisvc-controller-manager-6466c554d8-rlfbz\" (UID: \"ce3fa57a-f470-4d07-9cfc-1a3ece3df591\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"
Nov 03 19:58:29.450079 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.449903    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3fa57a-f470-4d07-9cfc-1a3ece3df591-cert\") pod \"llmisvc-controller-manager-6466c554d8-rlfbz\" (UID: \"ce3fa57a-f470-4d07-9cfc-1a3ece3df591\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"
Nov 03 19:58:29.451959 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.451929    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7f90cf-6e12-49ce-a541-95d5679edf9c-kube-api-access-w47hf" pod "ba7f90cf-6e12-49ce-a541-95d5679edf9c" (UID: "ba7f90cf-6e12-49ce-a541-95d5679edf9c"). InnerVolumeSpecName "kube-api-access-w47hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:29.550852 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.550808    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bff4q\" (UniqueName: \"kubernetes.io/projected/ce3fa57a-f470-4d07-9cfc-1a3ece3df591-kube-api-access-bff4q\") pod \"llmisvc-controller-manager-6466c554d8-rlfbz\" (UID: \"ce3fa57a-f470-4d07-9cfc-1a3ece3df591\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"
Nov 03 19:58:29.550852 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.550856    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3fa57a-f470-4d07-9cfc-1a3ece3df591-cert\") pod \"llmisvc-controller-manager-6466c554d8-rlfbz\" (UID: \"ce3fa57a-f470-4d07-9cfc-1a3ece3df591\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"
Nov 03 19:58:29.551476 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.550923    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w47hf\" (UniqueName: \"kubernetes.io/projected/ba7f90cf-6e12-49ce-a541-95d5679edf9c-kube-api-access-w47hf\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:29.553418 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.553400    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce3fa57a-f470-4d07-9cfc-1a3ece3df591-cert\") pod \"llmisvc-controller-manager-6466c554d8-rlfbz\" (UID: \"ce3fa57a-f470-4d07-9cfc-1a3ece3df591\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"
Nov 03 19:58:29.558874 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.558850    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bff4q\" (UniqueName: \"kubernetes.io/projected/ce3fa57a-f470-4d07-9cfc-1a3ece3df591-kube-api-access-bff4q\") pod \"llmisvc-controller-manager-6466c554d8-rlfbz\" (UID: \"ce3fa57a-f470-4d07-9cfc-1a3ece3df591\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"
Nov 03 19:58:29.678167 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.678138    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"
Nov 03 19:58:29.723006 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.722980    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh"
Nov 03 19:58:29.723177 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.723009    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-8t2jh" event={"ID":"ba7f90cf-6e12-49ce-a541-95d5679edf9c","Type":"ContainerDied","Data":"caf44e0a98534f0a930b670e7654694724363bbfb313cf40c125a0bcf53f9c5a"}
Nov 03 19:58:29.723177 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.723060    5600 scope.go:122] "RemoveContainer" containerID="8202be8683c0fce0141841d09aaa6bd20313d2c0498150c9704ff724560ece40"
Nov 03 19:58:29.753046 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.753018    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8t2jh"]
Nov 03 19:58:29.755638 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.755612    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-8t2jh"]
Nov 03 19:58:29.991852 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:29.991830    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"]
Nov 03 19:58:29.994151 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:58:29.994122    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3fa57a_f470_4d07_9cfc_1a3ece3df591.slice/crio-6ba02824df168b5ae7c0b70065c48b6984ae1a5ba241fa98ccc88e6925e4f0db WatchSource:0}: Error finding container 6ba02824df168b5ae7c0b70065c48b6984ae1a5ba241fa98ccc88e6925e4f0db: Status 404 returned error can't find the container with id 6ba02824df168b5ae7c0b70065c48b6984ae1a5ba241fa98ccc88e6925e4f0db
Nov 03 19:58:30.729992 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:30.729954    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz" event={"ID":"ce3fa57a-f470-4d07-9cfc-1a3ece3df591","Type":"ContainerStarted","Data":"a22d0df18264e76b855e99246e730e1b64c971acb23430e96e2cb0c0d4c14c3c"}
Nov 03 19:58:30.730395 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:30.730007    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz" event={"ID":"ce3fa57a-f470-4d07-9cfc-1a3ece3df591","Type":"ContainerStarted","Data":"6ba02824df168b5ae7c0b70065c48b6984ae1a5ba241fa98ccc88e6925e4f0db"}
Nov 03 19:58:30.730395 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:30.730090    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"
Nov 03 19:58:30.746222 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:30.746173    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz" podStartSLOduration=1.401762102 podStartE2EDuration="1.746158859s" totalImagesPullingTime="344.396757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:30.744674384 +0000 UTC m=+1093.257496166" watchObservedRunningTime="2026-11-03 19:58:30.746158859 +0000 UTC m=+1093.258980641"
Nov 03 19:58:31.614893 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:31.614858    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7f90cf-6e12-49ce-a541-95d5679edf9c" path="/var/lib/kubelet/pods/ba7f90cf-6e12-49ce-a541-95d5679edf9c/volumes"
Nov 03 19:58:35.703525 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:35.703491    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"]
Nov 03 19:58:35.704031 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:35.703694    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g" podUID="548f11a1-81b3-4b10-83a7-05e320e509b2" containerName="maas-api" containerID="cri-o://c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf" gracePeriod=30
Nov 03 19:58:35.948511 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:35.948479    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:36.105211 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.105106    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548f11a1-81b3-4b10-83a7-05e320e509b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548f11a1-81b3-4b10-83a7-05e320e509b2-openshift-service-ca\") pod \"548f11a1-81b3-4b10-83a7-05e320e509b2\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") "
Nov 03 19:58:36.105211 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.105213    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-maas-api-tls\") pod \"548f11a1-81b3-4b10-83a7-05e320e509b2\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") "
Nov 03 19:58:36.105425 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.105235    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-metrics-certs\") pod \"548f11a1-81b3-4b10-83a7-05e320e509b2\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") "
Nov 03 19:58:36.105425 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.105285    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548f11a1-81b3-4b10-83a7-05e320e509b2-kube-api-access-bn4nx\" (UniqueName: \"kubernetes.io/projected/548f11a1-81b3-4b10-83a7-05e320e509b2-kube-api-access-bn4nx\") pod \"548f11a1-81b3-4b10-83a7-05e320e509b2\" (UID: \"548f11a1-81b3-4b10-83a7-05e320e509b2\") "
Nov 03 19:58:36.105619 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.105590    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548f11a1-81b3-4b10-83a7-05e320e509b2-openshift-service-ca" pod "548f11a1-81b3-4b10-83a7-05e320e509b2" (UID: "548f11a1-81b3-4b10-83a7-05e320e509b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:36.107778 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.107743    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-metrics-certs" pod "548f11a1-81b3-4b10-83a7-05e320e509b2" (UID: "548f11a1-81b3-4b10-83a7-05e320e509b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:36.107902 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.107794    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548f11a1-81b3-4b10-83a7-05e320e509b2-kube-api-access-bn4nx" pod "548f11a1-81b3-4b10-83a7-05e320e509b2" (UID: "548f11a1-81b3-4b10-83a7-05e320e509b2"). InnerVolumeSpecName "kube-api-access-bn4nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:36.107902 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.107829    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-maas-api-tls" pod "548f11a1-81b3-4b10-83a7-05e320e509b2" (UID: "548f11a1-81b3-4b10-83a7-05e320e509b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:36.206834 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.206787    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548f11a1-81b3-4b10-83a7-05e320e509b2-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:36.206834 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.206830    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:36.206834 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.206842    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548f11a1-81b3-4b10-83a7-05e320e509b2-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:36.206834 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.206851    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn4nx\" (UniqueName: \"kubernetes.io/projected/548f11a1-81b3-4b10-83a7-05e320e509b2-kube-api-access-bn4nx\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:36.759026 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.758996    5600 generic.go:356] "Generic (PLEG): container finished" podID="548f11a1-81b3-4b10-83a7-05e320e509b2" containerID="c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf" exitCode=0
Nov 03 19:58:36.759454 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.759041    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"
Nov 03 19:58:36.759454 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.759081    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g" event={"ID":"548f11a1-81b3-4b10-83a7-05e320e509b2","Type":"ContainerDied","Data":"c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf"}
Nov 03 19:58:36.759454 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.759134    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g" event={"ID":"548f11a1-81b3-4b10-83a7-05e320e509b2","Type":"ContainerDied","Data":"621112903d1a1a4c051368078132775228253baa141a28db12c96c1f1aeba25d"}
Nov 03 19:58:36.759454 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.759163    5600 scope.go:122] "RemoveContainer" containerID="c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf"
Nov 03 19:58:36.768859 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.768843    5600 scope.go:122] "RemoveContainer" containerID="c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf"
Nov 03 19:58:36.769126 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:58:36.769110    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf\": container with ID starting with c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf not found: ID does not exist" containerID="c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf"
Nov 03 19:58:36.769161 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.769136    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf"} err="failed to get container status \"c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf\": rpc error: code = NotFound desc = could not find container \"c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf\": container with ID starting with c38d4b3c90fa68b7fdc012b14177ed45bee02991fa53a914245ac5bb2f588bcf not found: ID does not exist"
Nov 03 19:58:36.779651 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.779632    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"]
Nov 03 19:58:36.783242 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:36.783223    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-z8r9g"]
Nov 03 19:58:37.614735 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:37.614703    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="548f11a1-81b3-4b10-83a7-05e320e509b2" path="/var/lib/kubelet/pods/548f11a1-81b3-4b10-83a7-05e320e509b2/volumes"
Nov 03 19:58:45.045362 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.045326    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"]
Nov 03 19:58:45.045875 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.045757    5600 cpu_manager.go:414] "containerMap: removing container" podUID="548f11a1-81b3-4b10-83a7-05e320e509b2" containerName="maas-api"
Nov 03 19:58:45.045875 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.045770    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548f11a1-81b3-4b10-83a7-05e320e509b2" containerName="maas-api"
Nov 03 19:58:45.045875 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.045840    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="548f11a1-81b3-4b10-83a7-05e320e509b2" containerName="maas-api"
Nov 03 19:58:45.050308 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.050288    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.054336 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.054303    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 19:58:45.063725 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.063692    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"]
Nov 03 19:58:45.085703 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.085673    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.085850 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.085714    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.085850 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.085802    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bbda282-0b8b-4133-825d-a962403ea7cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.085850 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.085844    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.086018 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.085874    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.086080 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.086013    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.086080 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.086048    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2ccq\" (UniqueName: \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-kube-api-access-x2ccq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.086190 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.086165    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.086233 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.086207    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.140600 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.140574    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dkm76\""
Nov 03 19:58:45.187315 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187284    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.187489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187325    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.187489 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187369    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bbda282-0b8b-4133-825d-a962403ea7cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.187621 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187485    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.187621 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187539    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.187621 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187599    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.187775 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187626    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2ccq\" (UniqueName: \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-kube-api-access-x2ccq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.187775 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187664    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.187775 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187691    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.187775 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187758    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.187972 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187796    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.187972 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.187943    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.188060 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.188039    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.188407 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.188388    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bbda282-0b8b-4133-825d-a962403ea7cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.189923 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.189901    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.190400 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.190381    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.196024 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.195998    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.196216 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.196196    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2ccq\" (UniqueName: \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-kube-api-access-x2ccq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-28w6h\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.365924 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.365832    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:45.644456 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.644401    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"]
Nov 03 19:58:45.646340 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:58:45.646307    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbda282_0b8b_4133_825d_a962403ea7cd.slice/crio-4f51e58cecd1de31e829f2ceb38eee02b20bf22c23de4b74e91934721d3e30af WatchSource:0}: Error finding container 4f51e58cecd1de31e829f2ceb38eee02b20bf22c23de4b74e91934721d3e30af: Status 404 returned error can't find the container with id 4f51e58cecd1de31e829f2ceb38eee02b20bf22c23de4b74e91934721d3e30af
Nov 03 19:58:45.809767 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:45.809729    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" event={"ID":"9bbda282-0b8b-4133-825d-a962403ea7cd","Type":"ContainerStarted","Data":"4f51e58cecd1de31e829f2ceb38eee02b20bf22c23de4b74e91934721d3e30af"}
Nov 03 19:58:48.781881 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:48.781852    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:58:48.782135 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:48.781919    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:58:48.782135 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:48.781947    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:58:49.833532 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:49.833492    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" event={"ID":"9bbda282-0b8b-4133-825d-a962403ea7cd","Type":"ContainerStarted","Data":"04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046"}
Nov 03 19:58:49.855087 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:49.855031    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" podStartSLOduration=1.722637245 podStartE2EDuration="4.85501561s" totalImagesPullingTime="3.132378365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:49.853108439 +0000 UTC m=+1112.365930221" watchObservedRunningTime="2026-11-03 19:58:49.85501561 +0000 UTC m=+1112.367837392"
Nov 03 19:58:50.366574 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:50.366542    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:50.368159 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:50.368132    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-28w6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:58:50.368279 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:50.368190    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 19:58:51.366654 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:51.366621    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-28w6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:58:51.367064 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:51.366689    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 19:58:52.366154 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:52.366119    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-28w6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:58:52.366340 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:52.366182    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 19:58:53.366888 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:53.366843    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-28w6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:58:53.367317 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:53.366900    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 19:58:54.366757 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:54.366722    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-28w6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:58:54.366938 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:54.366774    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 19:58:55.370567 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:55.370538    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:55.370949 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:55.370815    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:55.371755 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:58:55.371733    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 19:58:58.794565 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:58:58.794529    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:59:00.060877 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.060846    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"]
Nov 03 19:59:00.072386 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.072350    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.082993 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.082961    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"]
Nov 03 19:59:00.120913 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.120874    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.121096 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.120949    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.121096 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.121048    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cead9ddc-269e-4641-80a7-30b294529b02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.121096 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.121082    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c874g\" (UniqueName: \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-kube-api-access-c874g\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.121238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.121149    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cead9ddc-269e-4641-80a7-30b294529b02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.121238 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.121178    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.121338 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.121241    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.121374 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.121338    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.121411 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.121378    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.222802 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.222765    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.222802 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.222799    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cead9ddc-269e-4641-80a7-30b294529b02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223066 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.222818    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c874g\" (UniqueName: \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-kube-api-access-c874g\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223066 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.222849    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cead9ddc-269e-4641-80a7-30b294529b02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223066 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.223028    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223066 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.223067    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223294 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.223119    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223294 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.223138    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223294 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.223174    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223294 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.223232    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223525 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.223350    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223525 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.223401    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223599 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.223581    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.223647 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.223627    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cead9ddc-269e-4641-80a7-30b294529b02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.225654 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.225634    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cead9ddc-269e-4641-80a7-30b294529b02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.225833 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.225813    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.230665 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.230639    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.231024 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.231004    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c874g\" (UniqueName: \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-kube-api-access-c874g\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rrs66\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.390174 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.390142    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:00.669501 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.669469    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"]
Nov 03 19:59:00.670660 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:59:00.670631    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcead9ddc_269e_4641_80a7_30b294529b02.slice/crio-7a95b57914162d4969c93cb96abb61a7b5c86f65ea3e29701a5a0872dc5ddb6c WatchSource:0}: Error finding container 7a95b57914162d4969c93cb96abb61a7b5c86f65ea3e29701a5a0872dc5ddb6c: Status 404 returned error can't find the container with id 7a95b57914162d4969c93cb96abb61a7b5c86f65ea3e29701a5a0872dc5ddb6c
Nov 03 19:59:00.674355 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.674316    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:59:00.674451 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.674388    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:59:00.674451 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.674420    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:59:00.889777 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.889737    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" event={"ID":"cead9ddc-269e-4641-80a7-30b294529b02","Type":"ContainerStarted","Data":"d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed"}
Nov 03 19:59:00.889979 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.889795    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" event={"ID":"cead9ddc-269e-4641-80a7-30b294529b02","Type":"ContainerStarted","Data":"7a95b57914162d4969c93cb96abb61a7b5c86f65ea3e29701a5a0872dc5ddb6c"}
Nov 03 19:59:00.909045 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:00.908991    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" podStartSLOduration=0.908975998 podStartE2EDuration="908.975998ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:00.908113537 +0000 UTC m=+1123.420935312" watchObservedRunningTime="2026-11-03 19:59:00.908975998 +0000 UTC m=+1123.421797779"
Nov 03 19:59:01.398460 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:01.395884    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:01.403062 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:01.403031    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rrs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 19:59:01.403306 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:01.403277    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 19:59:01.737458 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:01.737361    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rlfbz"
Nov 03 19:59:01.805457 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:01.804741    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"]
Nov 03 19:59:01.805457 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:01.804987    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml" podUID="5ade14a9-328f-4aa0-ad17-3268d284d25e" containerName="manager" containerID="cri-o://93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef" gracePeriod=30
Nov 03 19:59:02.065203 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.065182    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:59:02.144823 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.144790    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ade14a9-328f-4aa0-ad17-3268d284d25e-cert\" (UniqueName: \"kubernetes.io/secret/5ade14a9-328f-4aa0-ad17-3268d284d25e-cert\") pod \"5ade14a9-328f-4aa0-ad17-3268d284d25e\" (UID: \"5ade14a9-328f-4aa0-ad17-3268d284d25e\") "
Nov 03 19:59:02.144982 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.144899    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ade14a9-328f-4aa0-ad17-3268d284d25e-kube-api-access-q52g9\" (UniqueName: \"kubernetes.io/projected/5ade14a9-328f-4aa0-ad17-3268d284d25e-kube-api-access-q52g9\") pod \"5ade14a9-328f-4aa0-ad17-3268d284d25e\" (UID: \"5ade14a9-328f-4aa0-ad17-3268d284d25e\") "
Nov 03 19:59:02.147217 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.147187    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ade14a9-328f-4aa0-ad17-3268d284d25e-cert" pod "5ade14a9-328f-4aa0-ad17-3268d284d25e" (UID: "5ade14a9-328f-4aa0-ad17-3268d284d25e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:59:02.147328 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.147226    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ade14a9-328f-4aa0-ad17-3268d284d25e-kube-api-access-q52g9" pod "5ade14a9-328f-4aa0-ad17-3268d284d25e" (UID: "5ade14a9-328f-4aa0-ad17-3268d284d25e"). InnerVolumeSpecName "kube-api-access-q52g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:02.246487 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.246421    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q52g9\" (UniqueName: \"kubernetes.io/projected/5ade14a9-328f-4aa0-ad17-3268d284d25e-kube-api-access-q52g9\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:02.246487 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.246482    5600 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ade14a9-328f-4aa0-ad17-3268d284d25e-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:02.399457 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.393838    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rrs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 19:59:02.399457 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.393901    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 19:59:02.918141 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.918110    5600 generic.go:356] "Generic (PLEG): container finished" podID="5ade14a9-328f-4aa0-ad17-3268d284d25e" containerID="93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef" exitCode=0
Nov 03 19:59:02.918318 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.918158    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"
Nov 03 19:59:02.918318 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.918194    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml" event={"ID":"5ade14a9-328f-4aa0-ad17-3268d284d25e","Type":"ContainerDied","Data":"93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef"}
Nov 03 19:59:02.918318 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.918257    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml" event={"ID":"5ade14a9-328f-4aa0-ad17-3268d284d25e","Type":"ContainerDied","Data":"fc2ac3fb13182adc8e1d75ed3ca7ff20b4e7094d2fdb6d70135924ceb6372a8a"}
Nov 03 19:59:02.918318 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.918281    5600 scope.go:122] "RemoveContainer" containerID="93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef"
Nov 03 19:59:02.927873 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.927856    5600 scope.go:122] "RemoveContainer" containerID="93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef"
Nov 03 19:59:02.928153 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:59:02.928130    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef\": container with ID starting with 93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef not found: ID does not exist" containerID="93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef"
Nov 03 19:59:02.928194 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.928164    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef"} err="failed to get container status \"93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef\": rpc error: code = NotFound desc = could not find container \"93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef\": container with ID starting with 93d7b18c892a3d4d217f8a96111baf74782f6922322369666a0821b771bde0ef not found: ID does not exist"
Nov 03 19:59:02.938725 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.938700    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"]
Nov 03 19:59:02.940701 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:02.940683    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5g8ml"]
Nov 03 19:59:03.392511 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:03.390812    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rrs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 19:59:03.392511 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:03.390866    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 19:59:03.615288 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:03.615256    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ade14a9-328f-4aa0-ad17-3268d284d25e" path="/var/lib/kubelet/pods/5ade14a9-328f-4aa0-ad17-3268d284d25e/volumes"
Nov 03 19:59:04.391397 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:04.390969    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rrs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 19:59:04.391397 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:04.391026    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 19:59:05.403461 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:05.402847    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rrs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 19:59:05.403461 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:05.402928    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 19:59:06.406587 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:06.406546    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rrs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 19:59:06.407067 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:06.406620    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 19:59:07.394719 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:07.394689    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:07.395063 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:07.395042    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:07.395656 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:07.395638    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 19:59:15.075570 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.075535    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"]
Nov 03 19:59:15.075998 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.075937    5600 cpu_manager.go:414] "containerMap: removing container" podUID="5ade14a9-328f-4aa0-ad17-3268d284d25e" containerName="manager"
Nov 03 19:59:15.075998 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.075950    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ade14a9-328f-4aa0-ad17-3268d284d25e" containerName="manager"
Nov 03 19:59:15.076100 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.076046    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ade14a9-328f-4aa0-ad17-3268d284d25e" containerName="manager"
Nov 03 19:59:15.080723 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.080699    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.094059 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.094026    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"]
Nov 03 19:59:15.180248 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.180214    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.180248 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.180253    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.180562 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.180360    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.180562 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.180383    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.180562 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.180412    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea46236-0353-41e5-9382-9408c9398b00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.180710 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.180580    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.180710 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.180615    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.180710 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.180640    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x247v\" (UniqueName: \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-kube-api-access-x247v\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.180807 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.180714    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea46236-0353-41e5-9382-9408c9398b00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.281298 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.281259    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.281298 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.281301    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.281592 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.281344    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea46236-0353-41e5-9382-9408c9398b00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.281592 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.281399    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.281592 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.281472    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.281592 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.281497    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x247v\" (UniqueName: \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-kube-api-access-x247v\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.281592 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.281546    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea46236-0353-41e5-9382-9408c9398b00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.281592 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.281587    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.281862 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.281613    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.281926 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.281903    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.281988 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.281940    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.282336 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.282290    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.282452 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.282352    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.282716 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.282688    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea46236-0353-41e5-9382-9408c9398b00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.284001 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.283963    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.284324 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.284304    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea46236-0353-41e5-9382-9408c9398b00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.290742 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.290713    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.290906 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.290839    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x247v\" (UniqueName: \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-kube-api-access-x247v\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sk7xk\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.394995 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.394946    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:15.676474 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.676265    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"]
Nov 03 19:59:15.679219 ip-10-0-128-24 kubenswrapper[5600]: W1103 19:59:15.679190    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea46236_0353_41e5_9382_9408c9398b00.slice/crio-c417d9b73e9db3587deb35cd5b84e0fca3b0c2e2c32a2b8d76d791afb6963748 WatchSource:0}: Error finding container c417d9b73e9db3587deb35cd5b84e0fca3b0c2e2c32a2b8d76d791afb6963748: Status 404 returned error can't find the container with id c417d9b73e9db3587deb35cd5b84e0fca3b0c2e2c32a2b8d76d791afb6963748
Nov 03 19:59:15.682556 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.682518    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:59:15.682654 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.682597    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:59:15.682654 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.682639    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 19:59:15.986756 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.986666    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" event={"ID":"aea46236-0353-41e5-9382-9408c9398b00","Type":"ContainerStarted","Data":"d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6"}
Nov 03 19:59:15.986756 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:15.986728    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" event={"ID":"aea46236-0353-41e5-9382-9408c9398b00","Type":"ContainerStarted","Data":"c417d9b73e9db3587deb35cd5b84e0fca3b0c2e2c32a2b8d76d791afb6963748"}
Nov 03 19:59:16.009761 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:16.009686    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" podStartSLOduration=1.009666021 podStartE2EDuration="1.009666021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:16.006997594 +0000 UTC m=+1138.519819368" watchObservedRunningTime="2026-11-03 19:59:16.009666021 +0000 UTC m=+1138.522487804"
Nov 03 19:59:16.401617 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:16.401518    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:16.406637 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:16.406599    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sk7xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 19:59:16.406863 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:16.406667    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 19:59:17.396968 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:17.396926    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sk7xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 19:59:17.397283 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:17.397251    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 19:59:18.406638 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:18.406592    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sk7xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 19:59:18.407041 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:18.406663    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 19:59:19.402910 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:19.402874    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sk7xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 19:59:19.403217 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:19.403188    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 19:59:20.396461 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:20.395844    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sk7xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 19:59:20.396461 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:20.395925    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 19:59:21.402458 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:21.396849    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sk7xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 19:59:21.402458 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:21.396912    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 19:59:22.399083 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:22.399052    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:22.399354 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:22.399333    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:22.400140 ip-10-0-128-24 kubenswrapper[5600]: I1103 19:59:22.400125    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 19:59:28.810907 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:59:28.810852    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 19:59:58.796859 ip-10-0-128-24 kubenswrapper[5600]: E1103 19:59:58.796738    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:00:28.796660 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:00:28.796625    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:00:52.190111 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.188507    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"]
Nov 03 20:00:52.192958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.192934    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.198148 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.198122    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ab1e46-openshift-default-dockercfg-v2sht\""
Nov 03 20:00:52.212496 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.212467    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"]
Nov 03 20:00:52.325217 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.325185    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.325217 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.325225    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.325487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.325245    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmlhd\" (UniqueName: \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-kube-api-access-pmlhd\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.325487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.325266    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.325487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.325382    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.325487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.325419    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.325487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.325451    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.325671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.325537    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0e925b-4187-4596-8098-6a660bdb13a8-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.325671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.325565    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.427268 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.427235    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.427268 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.427275    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmlhd\" (UniqueName: \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-kube-api-access-pmlhd\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.428382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.427635    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.428382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.427690    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.428382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.427762    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.428382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.427797    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.428382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.427845    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0e925b-4187-4596-8098-6a660bdb13a8-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.428382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.427869    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.428382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.427968    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.428382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.428319    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.428382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.428335    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.430469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.430157    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0e925b-4187-4596-8098-6a660bdb13a8-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.430469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.431137    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.434502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.434356    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.434946 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.434919    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.435216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.435196    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.437923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.437899    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.439243 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.439218    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmlhd\" (UniqueName: \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-kube-api-access-pmlhd\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.513359 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.513269    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:52.805187 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.805162    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"]
Nov 03 20:00:52.808317 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:00:52.808272    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0e925b_4187_4596_8098_6a660bdb13a8.slice/crio-f819442cdb78bbd3c84298187c04ae83a5eb4cb6ff43bc9e0bc7fb798ec4c3ab WatchSource:0}: Error finding container f819442cdb78bbd3c84298187c04ae83a5eb4cb6ff43bc9e0bc7fb798ec4c3ab: Status 404 returned error can't find the container with id f819442cdb78bbd3c84298187c04ae83a5eb4cb6ff43bc9e0bc7fb798ec4c3ab
Nov 03 20:00:52.811052 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.811018    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:00:52.811144 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.811107    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:00:52.811199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:52.811148    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:00:53.653545 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:53.653507    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" event={"ID":"4e0e925b-4187-4596-8098-6a660bdb13a8","Type":"ContainerStarted","Data":"29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0"}
Nov 03 20:00:53.654247 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:53.654222    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" event={"ID":"4e0e925b-4187-4596-8098-6a660bdb13a8","Type":"ContainerStarted","Data":"f819442cdb78bbd3c84298187c04ae83a5eb4cb6ff43bc9e0bc7fb798ec4c3ab"}
Nov 03 20:00:53.677948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:53.677882    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" podStartSLOduration=1.677862981 podStartE2EDuration="1.677862981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:53.674313322 +0000 UTC m=+1236.187135115" watchObservedRunningTime="2026-11-03 20:00:53.677862981 +0000 UTC m=+1236.190684762"
Nov 03 20:00:54.514316 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:54.514281    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:54.520970 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:54.520938    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:54.659149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:54.659092    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:54.660156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:54.660132    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:00:58.359204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.359163    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"]
Nov 03 20:00:58.363123 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.363101    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:00:58.375938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.375500    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b42aea-dockercfg-t77xb\""
Nov 03 20:00:58.378006 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.377980    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"]
Nov 03 20:00:58.426222 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.426192    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b42aea\""
Nov 03 20:00:58.488483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.488448    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9nzg\" (UniqueName: \"kubernetes.io/projected/4683c83d-c95b-4878-9802-81982c339032-kube-api-access-z9nzg\") pod \"payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj\" (UID: \"4683c83d-c95b-4878-9802-81982c339032\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:00:58.488680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.488587    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4683c83d-c95b-4878-9802-81982c339032-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj\" (UID: \"4683c83d-c95b-4878-9802-81982c339032\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:00:58.590158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.590120    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9nzg\" (UniqueName: \"kubernetes.io/projected/4683c83d-c95b-4878-9802-81982c339032-kube-api-access-z9nzg\") pod \"payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj\" (UID: \"4683c83d-c95b-4878-9802-81982c339032\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:00:58.590365 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.590202    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4683c83d-c95b-4878-9802-81982c339032-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj\" (UID: \"4683c83d-c95b-4878-9802-81982c339032\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:00:58.591049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.591027    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4683c83d-c95b-4878-9802-81982c339032-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj\" (UID: \"4683c83d-c95b-4878-9802-81982c339032\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:00:58.600023 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.599995    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9nzg\" (UniqueName: \"kubernetes.io/projected/4683c83d-c95b-4878-9802-81982c339032-kube-api-access-z9nzg\") pod \"payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj\" (UID: \"4683c83d-c95b-4878-9802-81982c339032\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:00:58.680160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.680129    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:00:58.796919 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:00:58.796883    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:00:58.829538 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.828549    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"]
Nov 03 20:00:58.833880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.833852    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:58.844805 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.844764    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:00:58.844805 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.844772    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-0ed729-serving-cert\""
Nov 03 20:00:58.845172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.845124    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7bzpm\""
Nov 03 20:00:58.848008 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.847988    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"]
Nov 03 20:00:58.956465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.956409    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"]
Nov 03 20:00:58.959072 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:00:58.959045    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4683c83d_c95b_4878_9802_81982c339032.slice/crio-70e0c511fea82d481e871d31fe95a058da1faaaeac3076760c20e5f27c344cbc WatchSource:0}: Error finding container 70e0c511fea82d481e871d31fe95a058da1faaaeac3076760c20e5f27c344cbc: Status 404 returned error can't find the container with id 70e0c511fea82d481e871d31fe95a058da1faaaeac3076760c20e5f27c344cbc
Nov 03 20:00:58.994686 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.994656    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:58.994686 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.994695    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-metrics-certs\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:58.994987 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.994765    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:58.994987 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:58.994839    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dlzn\" (UniqueName: \"kubernetes.io/projected/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-kube-api-access-6dlzn\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:59.095717 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.095679    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:59.095904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.095754    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlzn\" (UniqueName: \"kubernetes.io/projected/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-kube-api-access-6dlzn\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:59.095904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.095816    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:59.095904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.095837    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-metrics-certs\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:59.096586 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.096512    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:59.098506 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.098476    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:59.098613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.098528    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-metrics-certs\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:59.104254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.104230    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlzn\" (UniqueName: \"kubernetes.io/projected/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-kube-api-access-6dlzn\") pod \"maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:59.151333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.151294    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:00:59.445837 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.445809    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"]
Nov 03 20:00:59.447638 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:00:59.447609    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d428d2f_5d0d_4fab_94d8_5efe40e8e49a.slice/crio-db8a4598f3cf0e79831cc1185505260697cda4faf336102e970265a7ab9a44be WatchSource:0}: Error finding container db8a4598f3cf0e79831cc1185505260697cda4faf336102e970265a7ab9a44be: Status 404 returned error can't find the container with id db8a4598f3cf0e79831cc1185505260697cda4faf336102e970265a7ab9a44be
Nov 03 20:00:59.692758 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.692718    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj" event={"ID":"4683c83d-c95b-4878-9802-81982c339032","Type":"ContainerStarted","Data":"70e0c511fea82d481e871d31fe95a058da1faaaeac3076760c20e5f27c344cbc"}
Nov 03 20:00:59.694463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:00:59.694398    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs" event={"ID":"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a","Type":"ContainerStarted","Data":"db8a4598f3cf0e79831cc1185505260697cda4faf336102e970265a7ab9a44be"}
Nov 03 20:01:00.043615 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.043573    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"]
Nov 03 20:01:00.047799 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.047778    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.056019 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.055987    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"]
Nov 03 20:01:00.057998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.057943    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-ab1e46-serving-cert\""
Nov 03 20:01:00.221994 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.221904    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c625f\" (UniqueName: \"kubernetes.io/projected/63df7cb9-548d-40c6-ab66-db4bb639501f-kube-api-access-c625f\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.222172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.222008    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63df7cb9-548d-40c6-ab66-db4bb639501f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.222172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.222076    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-metrics-certs\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.222172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.222119    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.322833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.322798    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-metrics-certs\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.322833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.322845    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.323101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.322930    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c625f\" (UniqueName: \"kubernetes.io/projected/63df7cb9-548d-40c6-ab66-db4bb639501f-kube-api-access-c625f\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.323101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.322985    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63df7cb9-548d-40c6-ab66-db4bb639501f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.323814 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.323787    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63df7cb9-548d-40c6-ab66-db4bb639501f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.326967 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.326919    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-metrics-certs\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.336046 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.336018    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c625f\" (UniqueName: \"kubernetes.io/projected/63df7cb9-548d-40c6-ab66-db4bb639501f-kube-api-access-c625f\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.336889 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.336865    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.384463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.384196    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:00.722261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.706480    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"]
Nov 03 20:01:00.722261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.708615    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs" event={"ID":"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a","Type":"ContainerStarted","Data":"d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a"}
Nov 03 20:01:00.722261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.709566    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:01:00.739763 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:00.731575    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs" podStartSLOduration=2.285643174 podStartE2EDuration="2.731555111s" totalImagesPullingTime="445.911937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:00.729133275 +0000 UTC m=+1243.241955057" watchObservedRunningTime="2026-11-03 20:01:00.731555111 +0000 UTC m=+1243.244376892"
Nov 03 20:01:01.716561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:01.716520    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj" event={"ID":"4683c83d-c95b-4878-9802-81982c339032","Type":"ContainerStarted","Data":"4b9f8de91739b7bb52f6a61c41ceb7d831c154669454ff0c831729e7a0de21d5"}
Nov 03 20:01:01.716780 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:01.716578    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:01:01.718428 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:01.718397    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn" event={"ID":"63df7cb9-548d-40c6-ab66-db4bb639501f","Type":"ContainerStarted","Data":"ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781"}
Nov 03 20:01:01.718571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:01.718474    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:01.718571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:01.718496    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn" event={"ID":"63df7cb9-548d-40c6-ab66-db4bb639501f","Type":"ContainerStarted","Data":"aad78c6e2f19042cca87718fd5d4c908afd8d9b48b89511afb04456a99d7fcc6"}
Nov 03 20:01:01.732210 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:01.732159    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj" podStartSLOduration=1.1401026 podStartE2EDuration="3.732139337s" totalImagesPullingTime="2.592036737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:01.73128347 +0000 UTC m=+1244.244105258" watchObservedRunningTime="2026-11-03 20:01:01.732139337 +0000 UTC m=+1244.244961117"
Nov 03 20:01:01.749724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:01.749674    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn" podStartSLOduration=1.3191390969999999 podStartE2EDuration="1.749660997s" totalImagesPullingTime="430.5219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:01.748490904 +0000 UTC m=+1244.261312685" watchObservedRunningTime="2026-11-03 20:01:01.749660997 +0000 UTC m=+1244.262482778"
Nov 03 20:01:03.036504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.036466    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"]
Nov 03 20:01:03.040324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.040302    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.049114 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.049092    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-5348ca-serving-cert\""
Nov 03 20:01:03.052448 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.052203    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"]
Nov 03 20:01:03.154533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.154491    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-metrics-certs\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.154728 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.154566    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.154728 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.154676    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93ba2d7-918c-45c3-b5ca-67803135937c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.154728 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.154703    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxvvd\" (UniqueName: \"kubernetes.io/projected/c93ba2d7-918c-45c3-b5ca-67803135937c-kube-api-access-fxvvd\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.255997 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.255956    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93ba2d7-918c-45c3-b5ca-67803135937c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.255997 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.256000    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxvvd\" (UniqueName: \"kubernetes.io/projected/c93ba2d7-918c-45c3-b5ca-67803135937c-kube-api-access-fxvvd\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.256248 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.256088    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-metrics-certs\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.256248 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.256146    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.256631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.256606    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93ba2d7-918c-45c3-b5ca-67803135937c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.258773 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.258753    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-metrics-certs\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.258882 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.258861    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.264507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.264487    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxvvd\" (UniqueName: \"kubernetes.io/projected/c93ba2d7-918c-45c3-b5ca-67803135937c-kube-api-access-fxvvd\") pod \"maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.358907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.358812    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:03.645475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.645446    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"]
Nov 03 20:01:03.647701 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:01:03.647662    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93ba2d7_918c_45c3_b5ca_67803135937c.slice/crio-94441590a3c72e1a7f77de728f4229c88088d19b05d0ab75eec060c07cf62b3c WatchSource:0}: Error finding container 94441590a3c72e1a7f77de728f4229c88088d19b05d0ab75eec060c07cf62b3c: Status 404 returned error can't find the container with id 94441590a3c72e1a7f77de728f4229c88088d19b05d0ab75eec060c07cf62b3c
Nov 03 20:01:03.735031 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:03.734985    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7" event={"ID":"c93ba2d7-918c-45c3-b5ca-67803135937c","Type":"ContainerStarted","Data":"94441590a3c72e1a7f77de728f4229c88088d19b05d0ab75eec060c07cf62b3c"}
Nov 03 20:01:04.742150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:04.742117    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7" event={"ID":"c93ba2d7-918c-45c3-b5ca-67803135937c","Type":"ContainerStarted","Data":"da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea"}
Nov 03 20:01:04.742607 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:04.742247    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:04.763654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:04.763539    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7" podStartSLOduration=1.271095387 podStartE2EDuration="1.763499517s" totalImagesPullingTime="492.40413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:04.761834587 +0000 UTC m=+1247.274656370" watchObservedRunningTime="2026-11-03 20:01:04.763499517 +0000 UTC m=+1247.276342377"
Nov 03 20:01:05.508106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.508069    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"]
Nov 03 20:01:05.518459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.517971    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.524585 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.524551    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"]
Nov 03 20:01:05.524919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.524894    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-b87733-serving-cert\""
Nov 03 20:01:05.602891 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.602859    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176d8e7c-0897-4a2a-bb67-cda280625d1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.603180 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.603161    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.603307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.603293    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-metrics-certs\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.603542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.603524    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2zz4\" (UniqueName: \"kubernetes.io/projected/176d8e7c-0897-4a2a-bb67-cda280625d1e-kube-api-access-z2zz4\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.707969 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.707930    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2zz4\" (UniqueName: \"kubernetes.io/projected/176d8e7c-0897-4a2a-bb67-cda280625d1e-kube-api-access-z2zz4\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.708601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.708579    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176d8e7c-0897-4a2a-bb67-cda280625d1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.709574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.709544    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.710157 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.710137    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-metrics-certs\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.710719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.709450    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176d8e7c-0897-4a2a-bb67-cda280625d1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.713966 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.713943    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.719659 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.719639    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-metrics-certs\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.722406 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.722385    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2zz4\" (UniqueName: \"kubernetes.io/projected/176d8e7c-0897-4a2a-bb67-cda280625d1e-kube-api-access-z2zz4\") pod \"maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:05.842611 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:05.842527    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:06.173876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:06.173828    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"]
Nov 03 20:01:06.175879 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:01:06.175849    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176d8e7c_0897_4a2a_bb67_cda280625d1e.slice/crio-a0d98b33bec2e8c257260cc0233c46a5be574fbc3a941f2e269835db8087e6f7 WatchSource:0}: Error finding container a0d98b33bec2e8c257260cc0233c46a5be574fbc3a941f2e269835db8087e6f7: Status 404 returned error can't find the container with id a0d98b33bec2e8c257260cc0233c46a5be574fbc3a941f2e269835db8087e6f7
Nov 03 20:01:06.763387 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:06.763358    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph" event={"ID":"176d8e7c-0897-4a2a-bb67-cda280625d1e","Type":"ContainerStarted","Data":"a0d98b33bec2e8c257260cc0233c46a5be574fbc3a941f2e269835db8087e6f7"}
Nov 03 20:01:07.730498 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:07.730467    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:01:07.731788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:07.731766    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:01:07.775629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:07.774937    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph" event={"ID":"176d8e7c-0897-4a2a-bb67-cda280625d1e","Type":"ContainerStarted","Data":"ff76487e37c2631acc1a93cc5bf03c93372d531323058f3eb601437b2fe44736"}
Nov 03 20:01:07.775629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:07.775580    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:10.701651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:10.701590    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:01:10.702239 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:10.701679    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:10.755905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:10.755876    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:01:10.776874 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:10.776824    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph" podStartSLOduration=5.202411736 podStartE2EDuration="5.776810886s" totalImagesPullingTime="574.39915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:07.812393138 +0000 UTC m=+1250.325214920" watchObservedRunningTime="2026-11-03 20:01:10.776810886 +0000 UTC m=+1253.289632665"
Nov 03 20:01:12.728311 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:12.728234    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:01:12.830221 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:12.830145    5600 generic.go:356] "Generic (PLEG): container finished" podID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerID="29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0" exitCode=0
Nov 03 20:01:12.830415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:12.830221    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" event={"ID":"4e0e925b-4187-4596-8098-6a660bdb13a8","Type":"ContainerDied","Data":"29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0"}
Nov 03 20:01:12.830709 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:12.830693    5600 scope.go:122] "RemoveContainer" containerID="29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0"
Nov 03 20:01:12.832121 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:12.832097    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:01:12.832204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:12.832158    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:01:12.832204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:12.832189    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:01:13.838501 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:13.838468    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" event={"ID":"4e0e925b-4187-4596-8098-6a660bdb13a8","Type":"ContainerStarted","Data":"670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f"}
Nov 03 20:01:14.514458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:14.513579    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:01:14.514458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:14.513801    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 03 20:01:14.514458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:14.513869    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 03 20:01:14.830073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:14.829999    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:01:15.516879 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:15.516844    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 03 20:01:15.517553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:15.517509    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 03 20:01:16.516848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:16.516811    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 03 20:01:16.517155 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:16.517126    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 03 20:01:17.517459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:17.513829    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 03 20:01:17.517459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:17.513893    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 03 20:01:18.514518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:18.514476    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 03 20:01:18.514724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:18.514542    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 03 20:01:19.520981 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:19.520948    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:01:19.521481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:19.521353    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:01:19.522171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:19.522155    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:01:28.797918 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:01:28.797819    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:01:37.049427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.049395    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"]
Nov 03 20:01:37.054078 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.054057    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.060520 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.060491    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"default-dockercfg-9qptp\""
Nov 03 20:01:37.071228 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.070634    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"kube-root-ca.crt\""
Nov 03 20:01:37.071228 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.070797    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"openshift-service-ca.crt\""
Nov 03 20:01:37.074407 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.072676    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"simulator-w3-b87733-kserve-self-signed-certs\""
Nov 03 20:01:37.081776 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.081747    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"]
Nov 03 20:01:37.178284 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.178248    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-model-cache\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.178484 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.178301    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-tmp-dir\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.178484 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.178414    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-dshm\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.178484 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.178479    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-home\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.178658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.178564    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pfnn\" (UniqueName: \"kubernetes.io/projected/c544766f-9820-441d-9564-647d963fa1ea-kube-api-access-2pfnn\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.178658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.178628    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c544766f-9820-441d-9564-647d963fa1ea-tls-certs\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.280165 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.280130    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-home\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.280165 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.280167    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pfnn\" (UniqueName: \"kubernetes.io/projected/c544766f-9820-441d-9564-647d963fa1ea-kube-api-access-2pfnn\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.280422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.280193    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c544766f-9820-441d-9564-647d963fa1ea-tls-certs\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.280422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.280257    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-model-cache\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.280422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.280276    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-tmp-dir\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.280422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.280309    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-dshm\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.280696 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.280643    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-home\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.280854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.280669    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-model-cache\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.280854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.280734    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-tmp-dir\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.285449 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.285412    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-dshm\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.285568 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.285551    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c544766f-9820-441d-9564-647d963fa1ea-tls-certs\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.289039 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.289015    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pfnn\" (UniqueName: \"kubernetes.io/projected/c544766f-9820-441d-9564-647d963fa1ea-kube-api-access-2pfnn\") pod \"simulator-w3-b87733-kserve-65b6f56976-b6l7s\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") " pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.396940 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.396817    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:37.681162 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.681134    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"]
Nov 03 20:01:37.683259 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:01:37.683229    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc544766f_9820_441d_9564_647d963fa1ea.slice/crio-ca880659d5900c1c16f3c80140cff3ddee5bec7bdcdd73925170db68d18d5e26 WatchSource:0}: Error finding container ca880659d5900c1c16f3c80140cff3ddee5bec7bdcdd73925170db68d18d5e26: Status 404 returned error can't find the container with id ca880659d5900c1c16f3c80140cff3ddee5bec7bdcdd73925170db68d18d5e26
Nov 03 20:01:37.988583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:37.988493    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" event={"ID":"c544766f-9820-441d-9564-647d963fa1ea","Type":"ContainerStarted","Data":"ca880659d5900c1c16f3c80140cff3ddee5bec7bdcdd73925170db68d18d5e26"}
Nov 03 20:01:40.003677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:40.003637    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" event={"ID":"c544766f-9820-441d-9564-647d963fa1ea","Type":"ContainerStarted","Data":"ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7"}
Nov 03 20:01:40.021106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:40.021059    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podStartSLOduration=1.544399818 podStartE2EDuration="3.021046111s" totalImagesPullingTime="1.476646293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:40.019738854 +0000 UTC m=+1282.532560660" watchObservedRunningTime="2026-11-03 20:01:40.021046111 +0000 UTC m=+1282.533867927"
Nov 03 20:01:40.397682 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:40.397651    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:47.397192 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:47.397154    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:47.409823 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:47.409797    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:48.063836 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:48.063807    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:01:57.324682 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.324647    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"]
Nov 03 20:01:57.331898 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.331871    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.338033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.338006    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"]
Nov 03 20:01:57.340671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.340646    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-0ed729\"/\"openshift-service-ca.crt\""
Nov 03 20:01:57.343086 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.341705    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0ed729\"/\"default-dockercfg-pmtnk\""
Nov 03 20:01:57.343224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.341755    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0ed729\"/\"premium-w6-0ed729-kserve-self-signed-certs\""
Nov 03 20:01:57.343330 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.341773    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-0ed729\"/\"kube-root-ca.crt\""
Nov 03 20:01:57.392026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.391986    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-dshm\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.392210 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.392125    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrsn4\" (UniqueName: \"kubernetes.io/projected/427c352d-209c-43fd-9afd-0a0209d94893-kube-api-access-nrsn4\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.392210 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.392194    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-tmp-dir\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.392336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.392317    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-home\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.392389 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.392357    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-model-cache\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.392524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.392507    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427c352d-209c-43fd-9afd-0a0209d94893-tls-certs\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.493863 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.493813    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427c352d-209c-43fd-9afd-0a0209d94893-tls-certs\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.494085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.493900    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-dshm\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.494085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.493940    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrsn4\" (UniqueName: \"kubernetes.io/projected/427c352d-209c-43fd-9afd-0a0209d94893-kube-api-access-nrsn4\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.494085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.493986    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-tmp-dir\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.494085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.494037    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-home\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.494085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.494066    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-model-cache\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.494952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.494922    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-tmp-dir\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.495247 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.495223    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-home\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.495597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.495572    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-model-cache\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.498087 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.498044    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-dshm\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.498574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.498473    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427c352d-209c-43fd-9afd-0a0209d94893-tls-certs\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.504229 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.504197    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrsn4\" (UniqueName: \"kubernetes.io/projected/427c352d-209c-43fd-9afd-0a0209d94893-kube-api-access-nrsn4\") pod \"premium-w6-0ed729-kserve-745488b97c-5zxql\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") " pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.649715 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.649679    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:57.933519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:57.933391    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"]
Nov 03 20:01:57.937393 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:01:57.937361    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod427c352d_209c_43fd_9afd_0a0209d94893.slice/crio-c334d272a1ad477467e0ada227cd84adebefb119aae06169ad12fee419856934 WatchSource:0}: Error finding container c334d272a1ad477467e0ada227cd84adebefb119aae06169ad12fee419856934: Status 404 returned error can't find the container with id c334d272a1ad477467e0ada227cd84adebefb119aae06169ad12fee419856934
Nov 03 20:01:58.111199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:58.111158    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" event={"ID":"427c352d-209c-43fd-9afd-0a0209d94893","Type":"ContainerStarted","Data":"0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b"}
Nov 03 20:01:58.111199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:58.111205    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" event={"ID":"427c352d-209c-43fd-9afd-0a0209d94893","Type":"ContainerStarted","Data":"c334d272a1ad477467e0ada227cd84adebefb119aae06169ad12fee419856934"}
Nov 03 20:01:58.129746 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:58.129689    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podStartSLOduration=1.129669953 podStartE2EDuration="1.129669953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:58.126790253 +0000 UTC m=+1300.639612036" watchObservedRunningTime="2026-11-03 20:01:58.129669953 +0000 UTC m=+1300.642491733"
Nov 03 20:01:58.650508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:01:58.650472    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:01:58.795858 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:01:58.795820    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:02:07.650315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:07.650259    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:02:07.670261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:07.670221    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:02:08.187575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:08.187538    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:02:21.801515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.801478    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"]
Nov 03 20:02:21.805590 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.805564    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:21.809341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.809320    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-11a5b0-openshift-default-dockercfg-jmpd9\""
Nov 03 20:02:21.815880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.815852    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"]
Nov 03 20:02:21.916270 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.916232    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"]
Nov 03 20:02:21.920216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.920198    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:21.933008 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.932977    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"]
Nov 03 20:02:21.933181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.933160    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b42aea-openshift-default-dockercfg-7bhzv\""
Nov 03 20:02:21.953110 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.953077    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:21.953314 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.953130    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:21.953314 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.953151    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123af875-2394-48b6-baca-ef5056d5a6cf-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:21.953314 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.953281    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:21.953314 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.953305    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:21.953542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.953321    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:21.953542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.953344    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stfpj\" (UniqueName: \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-kube-api-access-stfpj\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:21.953542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.953378    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:21.953646 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.953598    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123af875-2394-48b6-baca-ef5056d5a6cf-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:21.986205 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.986163    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"]
Nov 03 20:02:21.990368 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:21.990340    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.003907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.003877    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"]
Nov 03 20:02:22.033114 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.033086    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-5348ca-openshift-default-dockercfg-jql62\""
Nov 03 20:02:22.055033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.054942    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123af875-2394-48b6-baca-ef5056d5a6cf-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.055033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.054984    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.055033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055021    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.055315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055046    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp4t2\" (UniqueName: \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-kube-api-access-kp4t2\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.055315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055133    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.055315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055224    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123af875-2394-48b6-baca-ef5056d5a6cf-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.055315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055259    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.055315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055282    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.055602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055426    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.055602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055533    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.055602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055584    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.055749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055615    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.055749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055638    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.055749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055655    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.055749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055672    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.055749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055688    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.055749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055712    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stfpj\" (UniqueName: \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-kube-api-access-stfpj\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.055749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055738    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.056077 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055754    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123af875-2394-48b6-baca-ef5056d5a6cf-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.056077 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055815    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.056077 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.055977    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.056227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.056190    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.056701 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.056668    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.058098 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.058078    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.058370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.058352    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123af875-2394-48b6-baca-ef5056d5a6cf-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.064023 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.063991    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.064138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.064042    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stfpj\" (UniqueName: \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-kube-api-access-stfpj\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.120963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.120923    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:22.156845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.156810    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.156845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.156849    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.157098 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.156890    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.157098 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.156919    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp4t2\" (UniqueName: \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-kube-api-access-kp4t2\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.157098 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.156961    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.157098 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157053    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.157098 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157094    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nk5t\" (UniqueName: \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-kube-api-access-8nk5t\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.157354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157131    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.157354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157161    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.157354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157192    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.157354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157229    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.157354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157257    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.157689 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157668    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.157745 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157711    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.157845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157826    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.157898 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157860    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.157951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157934    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.157998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157960    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.157998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157992    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.158094 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.157996    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.158094 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.158023    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.158673 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.158578    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.158816 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.158720    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.160138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.160114    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.160836 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.160807    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.169341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.169309    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp4t2\" (UniqueName: \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-kube-api-access-kp4t2\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.171099 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.170945    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.236796 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.236762    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:22.259346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.259312    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.259521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.259403    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.259521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.259424    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.259521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.259465    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.259680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.259519    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.259680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.259557    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.259680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.259597    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.259680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.259615    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.259680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.259632    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nk5t\" (UniqueName: \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-kube-api-access-8nk5t\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.259944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.259918    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.260215 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.260196    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.260580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.260527    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.260580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.260556    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.260920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.260894    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.262349 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.262327    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.264963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.264934    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.268853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.268833    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nk5t\" (UniqueName: \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-kube-api-access-8nk5t\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.269427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.269404    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.307469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.307363    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:22.415051 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.414952    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"]
Nov 03 20:02:22.419042 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:02:22.419008    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123af875_2394_48b6_baca_ef5056d5a6cf.slice/crio-a2798d7ac2ec96c69aa01fcbc761a261ae1207596a5a68e821e1e540c5043bc1 WatchSource:0}: Error finding container a2798d7ac2ec96c69aa01fcbc761a261ae1207596a5a68e821e1e540c5043bc1: Status 404 returned error can't find the container with id a2798d7ac2ec96c69aa01fcbc761a261ae1207596a5a68e821e1e540c5043bc1
Nov 03 20:02:22.421558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.421511    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:22.421688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.421590    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:22.421688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.421621    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:22.532706 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.532671    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"]
Nov 03 20:02:22.533625 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:02:22.533591    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca8cce9_d587_4466_b46d_3fa8bc0a8583.slice/crio-fd5fe91920699f91120cb78fed2b979d8f6a32c4d5130aa0c13f2776dc70a89b WatchSource:0}: Error finding container fd5fe91920699f91120cb78fed2b979d8f6a32c4d5130aa0c13f2776dc70a89b: Status 404 returned error can't find the container with id fd5fe91920699f91120cb78fed2b979d8f6a32c4d5130aa0c13f2776dc70a89b
Nov 03 20:02:22.537945 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.537897    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:22.538032 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.537990    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:22.538111 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.538093    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:22.615493 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.615461    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"]
Nov 03 20:02:22.620609 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:02:22.620576    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9a468e_01e6_45f1_b0df_4f1fe4c9f54a.slice/crio-36f11c91944d219c87b278176cc07ddfaf99d72c2099eee504ca8f962245b7ca WatchSource:0}: Error finding container 36f11c91944d219c87b278176cc07ddfaf99d72c2099eee504ca8f962245b7ca: Status 404 returned error can't find the container with id 36f11c91944d219c87b278176cc07ddfaf99d72c2099eee504ca8f962245b7ca
Nov 03 20:02:22.623524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.623488    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:22.623625 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.623555    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:22.623625 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:22.623583    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:23.275055 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:23.275013    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" event={"ID":"123af875-2394-48b6-baca-ef5056d5a6cf","Type":"ContainerStarted","Data":"0806e50cf74b495556ddf4ba64154dc2838425148861555fcdf10101b0f473fd"}
Nov 03 20:02:23.275778 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:23.275732    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" event={"ID":"123af875-2394-48b6-baca-ef5056d5a6cf","Type":"ContainerStarted","Data":"a2798d7ac2ec96c69aa01fcbc761a261ae1207596a5a68e821e1e540c5043bc1"}
Nov 03 20:02:23.280480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:23.280427    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" event={"ID":"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a","Type":"ContainerStarted","Data":"7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e"}
Nov 03 20:02:23.280687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:23.280671    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" event={"ID":"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a","Type":"ContainerStarted","Data":"36f11c91944d219c87b278176cc07ddfaf99d72c2099eee504ca8f962245b7ca"}
Nov 03 20:02:23.288858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:23.288828    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" event={"ID":"aca8cce9-d587-4466-b46d-3fa8bc0a8583","Type":"ContainerStarted","Data":"43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222"}
Nov 03 20:02:23.289109 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:23.289093    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" event={"ID":"aca8cce9-d587-4466-b46d-3fa8bc0a8583","Type":"ContainerStarted","Data":"fd5fe91920699f91120cb78fed2b979d8f6a32c4d5130aa0c13f2776dc70a89b"}
Nov 03 20:02:23.308005 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:23.307944    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" podStartSLOduration=2.307924198 podStartE2EDuration="2.307924198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:23.304413918 +0000 UTC m=+1325.817235701" watchObservedRunningTime="2026-11-03 20:02:23.307924198 +0000 UTC m=+1325.820745981"
Nov 03 20:02:23.311634 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:23.309657    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:23.321734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:23.309784    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:02:23.322030 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:23.321993    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:02:23.355101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:23.353579    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" podStartSLOduration=2.353545725 podStartE2EDuration="2.353545725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:23.352331097 +0000 UTC m=+1325.865152879" watchObservedRunningTime="2026-11-03 20:02:23.353545725 +0000 UTC m=+1325.866367507"
Nov 03 20:02:24.125684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:24.125643    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:24.127998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:24.127962    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:02:24.128230 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:24.128206    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:02:24.242553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:24.242505    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:24.261644 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:24.261606    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:02:24.261916 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:24.261890    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 20:02:24.350759 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:24.350727    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:02:24.351321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:24.351290    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:02:25.122457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:25.121844    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:02:25.122457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:25.121907    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:02:25.242597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:25.241826    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:02:25.242597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:25.241889    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 20:02:25.319961 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:25.319524    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:02:25.319961 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:25.319585    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:02:26.121969 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:26.121931    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:02:26.122669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:26.122616    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:02:26.238882 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:26.238847    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:02:26.239172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:26.239147    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 20:02:26.324456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:26.323370    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:02:26.324456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:26.323479    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:02:27.126478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:27.123901    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:02:27.126478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:27.123976    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:02:27.238866 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:27.238829    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:02:27.239491 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:27.239116    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 20:02:27.309462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:27.308822    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:02:27.309462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:27.308885    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:02:28.121492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:28.121458    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:02:28.121766 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:28.121740    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:02:28.244462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:28.243835    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:02:28.244462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:28.243894    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 20:02:28.308886 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:28.308850    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:02:28.309168 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:28.309138    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:02:28.827656 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:02:28.827588    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:02:29.122337 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:29.122237    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:02:29.122337 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:29.122314    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:02:29.238478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:29.237980    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:02:29.238478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:29.238043    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 20:02:29.308634 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:29.308587    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:02:29.309159 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:29.308660    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:02:30.125714 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:30.125676    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:30.126070 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:30.126048    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:30.126756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:30.126740    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:02:30.150395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:30.150301    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" podStartSLOduration=9.150274447 podStartE2EDuration="9.150274447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:23.378409588 +0000 UTC m=+1325.891231371" watchObservedRunningTime="2026-11-03 20:02:30.150274447 +0000 UTC m=+1332.663096228"
Nov 03 20:02:30.242019 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:30.241982    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:30.242261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:30.242237    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:30.242903 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:30.242888    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:02:30.313591 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:30.313558    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:30.314684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:30.314667    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:30.315278 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:30.315261    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:02:37.188936 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.188894    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"]
Nov 03 20:02:37.194151 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.194124    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.226799 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.226755    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"]
Nov 03 20:02:37.265004 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.264974    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-120fe3-openshift-default-dockercfg-7jshr\""
Nov 03 20:02:37.281639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.281602    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.281639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.281647    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbrv\" (UniqueName: \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-kube-api-access-mbbrv\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.281850 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.281763    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.281894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.281873    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.281928 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.281892    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.281961 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.281937    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.281992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.281961    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.282026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.282010    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.282091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.282071    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.383155 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383123    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbrv\" (UniqueName: \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-kube-api-access-mbbrv\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.383339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383175    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.383339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383233    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.383339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383256    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.383339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383288    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.383339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383321    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.383621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383360    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.383621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383400    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.383621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383452    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.383774 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383691    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.383774 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383751    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.384001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.383982    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.384092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.384068    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.384252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.384113    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.385759 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.385737    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.385956 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.385938    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.396485 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.396427    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbrv\" (UniqueName: \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-kube-api-access-mbbrv\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.396620 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.396532    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.510637 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.510554    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:37.798272 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.798240    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"]
Nov 03 20:02:37.800004 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:02:37.799975    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31b1b2f_fdf9_4bfd_a113_9ceb3503004d.slice/crio-e2ba2a7c1e9bfe266bcc0b0af4174c4d132c36f5b15999fcad1dc61f24442e97 WatchSource:0}: Error finding container e2ba2a7c1e9bfe266bcc0b0af4174c4d132c36f5b15999fcad1dc61f24442e97: Status 404 returned error can't find the container with id e2ba2a7c1e9bfe266bcc0b0af4174c4d132c36f5b15999fcad1dc61f24442e97
Nov 03 20:02:37.802785 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.802752    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:37.802903 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.802832    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:37.802903 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:37.802878    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:02:38.447576 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:38.447536    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" event={"ID":"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d","Type":"ContainerStarted","Data":"1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb"}
Nov 03 20:02:38.448046 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:38.447595    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" event={"ID":"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d","Type":"ContainerStarted","Data":"e2ba2a7c1e9bfe266bcc0b0af4174c4d132c36f5b15999fcad1dc61f24442e97"}
Nov 03 20:02:38.470470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:38.470383    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" podStartSLOduration=1.470362993 podStartE2EDuration="1.470362993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:38.468370394 +0000 UTC m=+1340.981192177" watchObservedRunningTime="2026-11-03 20:02:38.470362993 +0000 UTC m=+1340.983184769"
Nov 03 20:02:38.513882 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:38.513842    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:38.515339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:38.515308    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:02:38.515490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:38.515369    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 20:02:39.512032 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:39.511989    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:02:39.512497 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:39.512059    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 20:02:40.511737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:40.511702    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:02:40.511943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:40.511756    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 20:02:41.511019 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:41.510983    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:02:41.511550 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:41.511510    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 20:02:42.512842 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:42.512805    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:02:42.513283 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:42.512872    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 20:02:43.516935 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:43.516895    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:02:43.517628 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:43.517583    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 20:02:44.531598 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:44.531568    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:44.532053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:44.531942    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:44.533813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:02:44.533790    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:02:58.797611 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:02:58.797512    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:03:11.422615 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:11.422573    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-28w6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:03:11.423343 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:11.423310    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:17.733415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.733379    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"]
Nov 03 20:03:17.737409 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.737391    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:17.743888 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.743850    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:03:17.745725 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.745700    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:03:17.747342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.747319    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"]
Nov 03 20:03:17.758311 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.758273    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9806786-z59mf\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:17.758525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.758348    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcbpp\" (UniqueName: \"kubernetes.io/projected/ab39436f-a4a2-4b55-836e-d265ca01c0c1-kube-api-access-tcbpp\") pod \"maas-api-revoke-sub-b9806786-z59mf\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:17.758525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.758427    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9806786-z59mf\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:17.859092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.859055    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9806786-z59mf\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:17.859303 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.859109    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcbpp\" (UniqueName: \"kubernetes.io/projected/ab39436f-a4a2-4b55-836e-d265ca01c0c1-kube-api-access-tcbpp\") pod \"maas-api-revoke-sub-b9806786-z59mf\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:17.859303 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.859178    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9806786-z59mf\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:17.859887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.859865    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9806786-z59mf\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:17.861809 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.861782    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9806786-z59mf\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:17.871099 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:17.871070    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcbpp\" (UniqueName: \"kubernetes.io/projected/ab39436f-a4a2-4b55-836e-d265ca01c0c1-kube-api-access-tcbpp\") pod \"maas-api-revoke-sub-b9806786-z59mf\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:18.056758 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:18.056658    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:18.338822 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:18.338798    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"]
Nov 03 20:03:18.340495 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:03:18.340468    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab39436f_a4a2_4b55_836e_d265ca01c0c1.slice/crio-297a3fca039dcbce4a82563c6438d367dfa3d7372582f733278ac8d075c916c3 WatchSource:0}: Error finding container 297a3fca039dcbce4a82563c6438d367dfa3d7372582f733278ac8d075c916c3: Status 404 returned error can't find the container with id 297a3fca039dcbce4a82563c6438d367dfa3d7372582f733278ac8d075c916c3
Nov 03 20:03:18.342525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:18.342508    5600 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:03:18.739917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:18.739886    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf" event={"ID":"ab39436f-a4a2-4b55-836e-d265ca01c0c1","Type":"ContainerStarted","Data":"297a3fca039dcbce4a82563c6438d367dfa3d7372582f733278ac8d075c916c3"}
Nov 03 20:03:21.761751 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:21.761716    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf" event={"ID":"ab39436f-a4a2-4b55-836e-d265ca01c0c1","Type":"ContainerStarted","Data":"20b1c43f03132fdd210bf629d794eb5322bf045cc32de3ff4e5532ec4ec10025"}
Nov 03 20:03:21.780031 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:21.779961    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf" podStartSLOduration=1.757278793 podStartE2EDuration="4.779945545s" totalImagesPullingTime="3.022666752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:21.779796957 +0000 UTC m=+1384.292618751" watchObservedRunningTime="2026-11-03 20:03:21.779945545 +0000 UTC m=+1384.292767327"
Nov 03 20:03:22.004531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.004495    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"]
Nov 03 20:03:22.009897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.009866    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.032417 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.032381    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"]
Nov 03 20:03:22.092403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.092362    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"]
Nov 03 20:03:22.098338 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.097386    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.131467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.130519    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.131467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.130588    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.131467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.130637    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf21814-f278-425e-998e-a2448cb1fe12-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.131467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.130727    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.131467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.130765    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.131467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.130796    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf21814-f278-425e-998e-a2448cb1fe12-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.131467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.130910    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r74f\" (UniqueName: \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-kube-api-access-2r74f\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.131467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.130972    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.131467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.131010    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.137673 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.137629    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"]
Nov 03 20:03:22.231674 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.231638    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r74f\" (UniqueName: \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-kube-api-access-2r74f\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.231894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.231702    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.231894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.231733    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.231894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.231762    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.231894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.231792    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.231894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.231829    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf21814-f278-425e-998e-a2448cb1fe12-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.232168 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.231901    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.232168 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.231933    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.232168 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.231957    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf21814-f278-425e-998e-a2448cb1fe12-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.232329 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.232248    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.232329 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.232251    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.232329 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.232311    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.232846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.232818    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf21814-f278-425e-998e-a2448cb1fe12-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.233047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.233026    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.234470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.234411    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.234727 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.234712    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf21814-f278-425e-998e-a2448cb1fe12-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.253181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.253001    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.253181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.253019    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r74f\" (UniqueName: \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-kube-api-access-2r74f\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.333336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.333246    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.333550 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.333342    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.333550 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.333381    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.333550 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.333406    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.333724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.333592    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.333724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.333706    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psdpc\" (UniqueName: \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-kube-api-access-psdpc\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.333828 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.333796    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.333880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.333827    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.333933 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.333915    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.340983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.340953    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:22.436711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.435822    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psdpc\" (UniqueName: \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-kube-api-access-psdpc\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.436711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.435936    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.436711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.435965    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.436711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.436034    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.436711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.436067    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.436711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.436097    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.436711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.436134    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.436711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.436170    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.436711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.436246    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.437736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.437367    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.437736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.437460    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.437736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.437703    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.438071 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.437968    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.438144 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.438106    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.447160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.446910    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psdpc\" (UniqueName: \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-kube-api-access-psdpc\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.447160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.447062    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.447520 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.447356    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.448252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.448222    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.454376 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.454352    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:22.629100 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.629075    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"]
Nov 03 20:03:22.631694 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:03:22.631669    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf21814_f278_425e_998e_a2448cb1fe12.slice/crio-93be39bcd8ab643e2897ccffe7a3b4e65b87ff5f26fcafc8e56d65e0a145424c WatchSource:0}: Error finding container 93be39bcd8ab643e2897ccffe7a3b4e65b87ff5f26fcafc8e56d65e0a145424c: Status 404 returned error can't find the container with id 93be39bcd8ab643e2897ccffe7a3b4e65b87ff5f26fcafc8e56d65e0a145424c
Nov 03 20:03:22.634697 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.634667    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:22.634770 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.634736    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:22.634812 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.634790    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:22.758328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.758144    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"]
Nov 03 20:03:22.763387 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:03:22.761941    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92a40ff_88c1_4ba0_99cf_47b494fe2ebe.slice/crio-222c42ada4d72744ee89d131ee691be6122720d3073d85282a4e6c326362bb99 WatchSource:0}: Error finding container 222c42ada4d72744ee89d131ee691be6122720d3073d85282a4e6c326362bb99: Status 404 returned error can't find the container with id 222c42ada4d72744ee89d131ee691be6122720d3073d85282a4e6c326362bb99
Nov 03 20:03:22.765848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.765808    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:22.765965 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.765898    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:22.765965 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.765942    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:22.771146 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.771123    5600 generic.go:356] "Generic (PLEG): container finished" podID="ab39436f-a4a2-4b55-836e-d265ca01c0c1" containerID="20b1c43f03132fdd210bf629d794eb5322bf045cc32de3ff4e5532ec4ec10025" exitCode=0
Nov 03 20:03:22.771273 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.771211    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf" event={"ID":"ab39436f-a4a2-4b55-836e-d265ca01c0c1","Type":"ContainerDied","Data":"20b1c43f03132fdd210bf629d794eb5322bf045cc32de3ff4e5532ec4ec10025"}
Nov 03 20:03:22.773114 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.773031    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" event={"ID":"dcf21814-f278-425e-998e-a2448cb1fe12","Type":"ContainerStarted","Data":"dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a"}
Nov 03 20:03:22.773114 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.773084    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" event={"ID":"dcf21814-f278-425e-998e-a2448cb1fe12","Type":"ContainerStarted","Data":"93be39bcd8ab643e2897ccffe7a3b4e65b87ff5f26fcafc8e56d65e0a145424c"}
Nov 03 20:03:22.806556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:22.806499    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" podStartSLOduration=1.806477642 podStartE2EDuration="1.806477642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:22.804890456 +0000 UTC m=+1385.317712240" watchObservedRunningTime="2026-11-03 20:03:22.806477642 +0000 UTC m=+1385.319299424"
Nov 03 20:03:23.345473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:23.344491    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:23.346078 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:23.346052    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:23.346229 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:23.346103    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:03:23.798459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:23.793644    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" event={"ID":"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe","Type":"ContainerStarted","Data":"c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05"}
Nov 03 20:03:23.798459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:23.793705    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" event={"ID":"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe","Type":"ContainerStarted","Data":"222c42ada4d72744ee89d131ee691be6122720d3073d85282a4e6c326362bb99"}
Nov 03 20:03:23.829006 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:23.828938    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" podStartSLOduration=1.8289158460000001 podStartE2EDuration="1.828915846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:23.819880479 +0000 UTC m=+1386.332702261" watchObservedRunningTime="2026-11-03 20:03:23.828915846 +0000 UTC m=+1386.341737631"
Nov 03 20:03:24.381459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.380860    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:24.381459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.380920    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:03:24.460743 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.460573    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:24.460743 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.460669    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:03:24.461130 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.460717    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:03:24.606461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.603666    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:24.855151 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.855108    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-openshift-ca\") pod \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") "
Nov 03 20:03:24.867772 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.867742    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab39436f-a4a2-4b55-836e-d265ca01c0c1-kube-api-access-tcbpp\" (UniqueName: \"kubernetes.io/projected/ab39436f-a4a2-4b55-836e-d265ca01c0c1-kube-api-access-tcbpp\") pod \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") "
Nov 03 20:03:24.868116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.868089    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-xks-ca\") pod \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\" (UID: \"ab39436f-a4a2-4b55-836e-d265ca01c0c1\") "
Nov 03 20:03:24.887467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.880632    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"
Nov 03 20:03:24.889527 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.865699    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-openshift-ca" pod "ab39436f-a4a2-4b55-836e-d265ca01c0c1" (UID: "ab39436f-a4a2-4b55-836e-d265ca01c0c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:24.894508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.888026    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf" event={"ID":"ab39436f-a4a2-4b55-836e-d265ca01c0c1","Type":"ContainerDied","Data":"297a3fca039dcbce4a82563c6438d367dfa3d7372582f733278ac8d075c916c3"}
Nov 03 20:03:24.894829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.894794    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="297a3fca039dcbce4a82563c6438d367dfa3d7372582f733278ac8d075c916c3"
Nov 03 20:03:24.895729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.895701    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-xks-ca" pod "ab39436f-a4a2-4b55-836e-d265ca01c0c1" (UID: "ab39436f-a4a2-4b55-836e-d265ca01c0c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:03:24.906329 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.902389    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab39436f-a4a2-4b55-836e-d265ca01c0c1-kube-api-access-tcbpp" pod "ab39436f-a4a2-4b55-836e-d265ca01c0c1" (UID: "ab39436f-a4a2-4b55-836e-d265ca01c0c1"). InnerVolumeSpecName "kube-api-access-tcbpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:25.006475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.984342    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:25.006475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.984379    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcbpp\" (UniqueName: \"kubernetes.io/projected/ab39436f-a4a2-4b55-836e-d265ca01c0c1-kube-api-access-tcbpp\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:25.006475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:24.984398    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab39436f-a4a2-4b55-836e-d265ca01c0c1-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:25.343460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:25.343269    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:25.343460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:25.343329    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:03:25.460463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:25.456835    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:03:25.460463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:25.456896    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:03:26.356455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:26.343883    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:26.356455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:26.343962    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:03:26.484455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:26.474723    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:03:26.484455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:26.474795    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:03:27.367458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:27.364497    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:27.367458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:27.364595    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:03:27.458577 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:27.457075    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:03:27.458577 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:27.457146    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:03:28.344567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:28.344534    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:28.344846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:28.344820    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:03:28.455007 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:28.454969    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:03:28.455569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:28.455525    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:03:28.863068 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:03:28.863032    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:03:29.342262 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:29.341815    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:29.342262 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:29.341886    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:03:29.454869 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:29.454832    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:03:29.455172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:29.455141    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:03:30.347504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:30.344928    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:30.347504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:30.344989    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 03 20:03:30.455056 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:30.455012    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 03 20:03:30.455232 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:30.455134    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 03 20:03:32.383676 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:32.383637    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:03:32.384334 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:32.384297    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:32.442839 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:32.442801    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:32.444078 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:32.443960    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:32.446346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:32.446321    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:03:32.501224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:32.500606    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:03:32.501224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:32.500691    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:32.725720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:32.725615    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:32.726063 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:32.726033    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:32.730234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:32.730214    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:03:37.103569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.103535    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"]
Nov 03 20:03:37.104720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.104681    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ab39436f-a4a2-4b55-836e-d265ca01c0c1" containerName="revoke-keys"
Nov 03 20:03:37.104720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.104708    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab39436f-a4a2-4b55-836e-d265ca01c0c1" containerName="revoke-keys"
Nov 03 20:03:37.104925 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.104857    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab39436f-a4a2-4b55-836e-d265ca01c0c1" containerName="revoke-keys"
Nov 03 20:03:37.108784 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.108763    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.136450 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.136406    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"]
Nov 03 20:03:37.186139 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.186101    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.186139 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.186144    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.186393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.186175    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.186393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.186215    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbsnq\" (UniqueName: \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-kube-api-access-gbsnq\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.186393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.186250    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.186393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.186276    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.186393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.186315    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.186684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.186401    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.186684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.186467    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.205321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.205285    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"]
Nov 03 20:03:37.209181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.209160    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.212901 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.212877    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b87733-openshift-default-dockercfg-lvxzx\""
Nov 03 20:03:37.223968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.223943    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"]
Nov 03 20:03:37.287754 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.287714    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.287937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.287780    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.287937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.287818    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.287937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.287846    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.287937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.287920    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgckd\" (UniqueName: \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-kube-api-access-wgckd\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.288246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.287966    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.288246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288077    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.288246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288120    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.288246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288164    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbsnq\" (UniqueName: \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-kube-api-access-gbsnq\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.288246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288206    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.288246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288229    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.288528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288257    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.288528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288280    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.288528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288319    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.288528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288365    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.288528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288387    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.288904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288829    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.288904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.288880    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.289086 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.289005    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.289086 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.289025    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.289352 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.289329    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.289519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.289499    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.289940 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.289921    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.290899 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.290877    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.291042 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.291020    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.297233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.297213    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbsnq\" (UniqueName: \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-kube-api-access-gbsnq\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.297411 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.297389    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.362640 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.362556    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"]
Nov 03 20:03:37.367870 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.367844    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.376389 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.376315    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-0ed729-openshift-default-dockercfg-cflmg\""
Nov 03 20:03:37.382051 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.382020    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"]
Nov 03 20:03:37.391049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.390346    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.391049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.390391    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.391049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.390418    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.391049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.390537    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.391049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.390574    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.391049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.390673    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.391049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.390710    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgckd\" (UniqueName: \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-kube-api-access-wgckd\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.391049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.390738    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.391049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.390768    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.391049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.390806    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.391678 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.391092    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.391678 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.391387    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.393620 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.392831    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.395795 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.395367    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.406723 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.406668    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgckd\" (UniqueName: \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-kube-api-access-wgckd\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.406926 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.406794    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.421475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.418169    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.421475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.418425    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.438994 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.438959    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:37.492478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.492023    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.492478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.492085    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmbpg\" (UniqueName: \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-kube-api-access-gmbpg\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.492478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.492126    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.492478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.492155    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.492478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.492210    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.492478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.492236    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.492478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.492252    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.492478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.492281    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.492478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.492303    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.533235 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.528138    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.593928    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.594026    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.594064    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.594087    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.594122    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.594149    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.594182    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.594242    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbpg\" (UniqueName: \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-kube-api-access-gmbpg\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.594298    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.595016    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.595260    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.596044    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.596601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.596447    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.597949 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.596651    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.599766 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.599738    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.602776 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.602725    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.606942 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.606920    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.611484 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.610979    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmbpg\" (UniqueName: \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-kube-api-access-gmbpg\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.703871 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.703525    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:37.763177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.763139    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"]
Nov 03 20:03:37.772297 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.772264    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:37.772421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.772330    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:37.772421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.772363    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:37.876055 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.876001    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"]
Nov 03 20:03:37.959075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.958234    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:37.959075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.958331    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:37.959075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:37.958374    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:38.041112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.040454    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"]
Nov 03 20:03:38.044111 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:03:38.044053    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c872495_4d89_43cf_8936_2d2cc87e2fa2.slice/crio-d249c4907304f2aa51f1622465b0053fd902165cdc3ad788f80dbf211e0a7d79 WatchSource:0}: Error finding container d249c4907304f2aa51f1622465b0053fd902165cdc3ad788f80dbf211e0a7d79: Status 404 returned error can't find the container with id d249c4907304f2aa51f1622465b0053fd902165cdc3ad788f80dbf211e0a7d79
Nov 03 20:03:38.048624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.047937    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:38.048624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.048023    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:38.048624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.048063    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:38.107238 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.107143    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" event={"ID":"7c872495-4d89-43cf-8936-2d2cc87e2fa2","Type":"ContainerStarted","Data":"d249c4907304f2aa51f1622465b0053fd902165cdc3ad788f80dbf211e0a7d79"}
Nov 03 20:03:38.131093 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.131016    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" event={"ID":"6d0b2608-2249-49b3-a2bc-57aa8d01d739","Type":"ContainerStarted","Data":"3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0"}
Nov 03 20:03:38.137748 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.137618    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" event={"ID":"6d0b2608-2249-49b3-a2bc-57aa8d01d739","Type":"ContainerStarted","Data":"82cc4a633354ea7e145e8314b056bd8c5cafd936a62e8a680e90fd0503113651"}
Nov 03 20:03:38.138768 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.138688    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" event={"ID":"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf","Type":"ContainerStarted","Data":"5ac7cd21b046bb4e8b5623247df257f3919ec7c08b52e7c704f984ed67b2d079"}
Nov 03 20:03:38.138768 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.138741    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" event={"ID":"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf","Type":"ContainerStarted","Data":"ae67bf5a77f95de8b7bd4b7d3f9316d2389870a18569881c03a1f73117b05d2b"}
Nov 03 20:03:38.163350 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.160991    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podStartSLOduration=1.160972248 podStartE2EDuration="1.160972248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:38.156355672 +0000 UTC m=+1400.669177466" watchObservedRunningTime="2026-11-03 20:03:38.160972248 +0000 UTC m=+1400.673794033"
Nov 03 20:03:38.182229 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.182149    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podStartSLOduration=1.18213043 podStartE2EDuration="1.18213043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:38.180733245 +0000 UTC m=+1400.693555029" watchObservedRunningTime="2026-11-03 20:03:38.18213043 +0000 UTC m=+1400.694952214"
Nov 03 20:03:38.445456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.444598    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:38.445456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.444730    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:03:38.445456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.444777    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:03:38.529109 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:38.529072    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:39.199823 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:39.199785    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" event={"ID":"7c872495-4d89-43cf-8936-2d2cc87e2fa2","Type":"ContainerStarted","Data":"0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48"}
Nov 03 20:03:39.450066 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:39.449962    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:03:39.450361 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:39.450314    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:03:39.535461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:39.534139    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:39.535461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:39.534226    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:03:39.706394 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:39.706305    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:39.706730 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:39.706711    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:39.706852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:39.706832    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:03:40.443459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:40.442946    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:03:40.443459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:40.443024    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:03:40.541459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:40.540829    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:40.541459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:40.540907    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:03:40.711510 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:40.711357    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:40.711510 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:40.711414    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:03:41.442405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:41.441918    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:03:41.442405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:41.441985    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:03:41.537458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:41.534791    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:41.537458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:41.534866    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:03:41.705522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:41.704880    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:41.705522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:41.704951    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:03:42.441339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:42.440884    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:03:42.441339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:42.440951    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:03:42.538395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:42.537936    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:42.538395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:42.537995    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:03:42.706892 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:42.706787    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:42.707155 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:42.707120    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:03:43.439405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:43.439363    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:03:43.439699 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:43.439667    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:03:43.532003 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:43.531967    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:43.532293 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:43.532239    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:03:43.719013 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:43.718918    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:43.719649 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:43.719614    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:03:44.442015 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:44.441982    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:03:44.442314 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:44.442289    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:03:44.549742 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:44.549694    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:44.549918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:44.549790    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:03:44.721511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:44.718899    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:44.721511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:44.718975    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:03:45.446334 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:45.445869    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:03:45.446334 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:45.445945    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:03:45.530887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:45.530853    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:45.531159 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:45.531133    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:03:45.591190 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:45.589526    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:03:45.591190 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:45.589604    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:45.706697 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:45.705954    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:45.706697 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:45.706025    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:03:46.442456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:46.441846    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:03:46.442456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:46.441927    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:03:46.541674 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:46.541639    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:46.541944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:46.541914    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:03:46.722302 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:46.722210    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:46.722654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:46.722618    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:03:47.442031 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:47.441991    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:03:47.442392 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:47.442343    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:03:47.572500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:47.566101    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:47.572500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:47.566169    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:03:47.705535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:47.704860    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:47.705535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:47.704926    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:03:48.440505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:48.440021    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:03:48.440505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:48.440084    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:03:48.529074 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:48.529034    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:48.529302 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:48.529101    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 20:03:49.638694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:49.638661    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:49.640617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:49.640594    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:49.642149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:49.642095    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:03:49.663568 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:49.663516    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" podStartSLOduration=12.663500259 podStartE2EDuration="12.663500259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:39.245979099 +0000 UTC m=+1401.758800896" watchObservedRunningTime="2026-11-03 20:03:49.663500259 +0000 UTC m=+1412.176322040"
Nov 03 20:03:49.962540 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:49.962428    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:49.963043 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:49.963013    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:49.963860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:49.963837    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:03:50.288804 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:50.288648    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:50.289235 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:50.289209    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:50.290253 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:50.290228    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:03:52.221966 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.221933    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"]
Nov 03 20:03:52.227204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.227177    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.239234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.239208    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"]
Nov 03 20:03:52.336414 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.336377    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.336414 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.336421    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.336667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.336619    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.336737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.336721    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.336786 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.336746    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2878c757-c831-4404-9182-8bbb34d3274b-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.336833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.336788    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.336833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.336824    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2878c757-c831-4404-9182-8bbb34d3274b-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.336989 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.336969    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhcb\" (UniqueName: \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-kube-api-access-hlhcb\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.337051 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.337020    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.437813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.437782    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhcb\" (UniqueName: \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-kube-api-access-hlhcb\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.437825    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.437846    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.437871    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.437929    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.437968    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.437983    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2878c757-c831-4404-9182-8bbb34d3274b-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.438021    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438428 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.438044    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2878c757-c831-4404-9182-8bbb34d3274b-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.438459    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.438481    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438619 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.438551    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.438790 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.438736    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.439100 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.439012    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2878c757-c831-4404-9182-8bbb34d3274b-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.440871 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.440841    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2878c757-c831-4404-9182-8bbb34d3274b-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.441019 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.441003    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.443260 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.443237    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"]
Nov 03 20:03:52.449687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.449669    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.454115 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.454095    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhcb\" (UniqueName: \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-kube-api-access-hlhcb\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.455112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.455088    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.462860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.462837    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"]
Nov 03 20:03:52.538977 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.538947    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd86m\" (UniqueName: \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-kube-api-access-nd86m\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.539178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.538996    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460ffe79-e497-4547-8d0e-9db401527282-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.539178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.539087    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.539178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.539128    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460ffe79-e497-4547-8d0e-9db401527282-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.539293 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.539225    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.539343 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.539290    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.539377 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.539347    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.539511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.539487    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.539559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.539545    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.546455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.546418    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.647614    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.647703    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.647732    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.647803    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.647859    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.647925    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd86m\" (UniqueName: \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-kube-api-access-nd86m\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.647959    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460ffe79-e497-4547-8d0e-9db401527282-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.647992    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.648016    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460ffe79-e497-4547-8d0e-9db401527282-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.648831    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460ffe79-e497-4547-8d0e-9db401527282-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.649525    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.649739    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.651840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.650297    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.652768 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.652649    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.659745 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.659500    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.662268 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.662194    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd86m\" (UniqueName: \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-kube-api-access-nd86m\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.662897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.662855    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.665225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.664221    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460ffe79-e497-4547-8d0e-9db401527282-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.793842 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.793747    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:52.841930 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.841907    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"]
Nov 03 20:03:52.848253 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.848208    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:52.848380 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.848278    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:52.848380 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:52.848312    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:53.087209 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.087112    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"]
Nov 03 20:03:53.093710 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:03:53.092712    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460ffe79_e497_4547_8d0e_9db401527282.slice/crio-050cb464ad0992ef6ddb7aa429b3abf4757669fb6c913b5ebad497eb8943f272 WatchSource:0}: Error finding container 050cb464ad0992ef6ddb7aa429b3abf4757669fb6c913b5ebad497eb8943f272: Status 404 returned error can't find the container with id 050cb464ad0992ef6ddb7aa429b3abf4757669fb6c913b5ebad497eb8943f272
Nov 03 20:03:53.096019 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.095981    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:53.096130 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.096063    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:53.096130 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.096105    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:03:53.570189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.570157    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"]
Nov 03 20:03:53.578710 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.578682    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.591101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.591067    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"]
Nov 03 20:03:53.591339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.591317    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0ed729\"/\"distinct-w6-0ed729-kserve-self-signed-certs\""
Nov 03 20:03:53.608298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.599158    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" event={"ID":"2878c757-c831-4404-9182-8bbb34d3274b","Type":"ContainerStarted","Data":"021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f"}
Nov 03 20:03:53.608298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.599198    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" event={"ID":"2878c757-c831-4404-9182-8bbb34d3274b","Type":"ContainerStarted","Data":"9d38c416eca11baa9d4c1a1f556213844b3c49276b452bb2a0f4b2cd05cb2464"}
Nov 03 20:03:53.608298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.604149    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" event={"ID":"460ffe79-e497-4547-8d0e-9db401527282","Type":"ContainerStarted","Data":"f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4"}
Nov 03 20:03:53.608298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.604196    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" event={"ID":"460ffe79-e497-4547-8d0e-9db401527282","Type":"ContainerStarted","Data":"050cb464ad0992ef6ddb7aa429b3abf4757669fb6c913b5ebad497eb8943f272"}
Nov 03 20:03:53.643915 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.643862    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" podStartSLOduration=1.643848703 podStartE2EDuration="1.643848703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:53.640740273 +0000 UTC m=+1416.153562056" watchObservedRunningTime="2026-11-03 20:03:53.643848703 +0000 UTC m=+1416.156670485"
Nov 03 20:03:53.661017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.660974    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-dshm\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.661171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.661047    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-model-cache\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.661171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.661128    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-tmp-dir\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.661171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.661155    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x6r6\" (UniqueName: \"kubernetes.io/projected/424be2c2-94d5-42d9-b434-d2bdaa809e85-kube-api-access-4x6r6\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.661359 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.661197    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424be2c2-94d5-42d9-b434-d2bdaa809e85-tls-certs\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.661359 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.661221    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-home\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.668739 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.668673    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" podStartSLOduration=1.66865782 podStartE2EDuration="1.66865782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:53.663908859 +0000 UTC m=+1416.176730641" watchObservedRunningTime="2026-11-03 20:03:53.66865782 +0000 UTC m=+1416.181479655"
Nov 03 20:03:53.763582 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.763220    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424be2c2-94d5-42d9-b434-d2bdaa809e85-tls-certs\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.763582 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.763271    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-home\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.763582 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.763466    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-dshm\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.763582 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.763550    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-model-cache\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.763941 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.763626    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-tmp-dir\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.763941 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.763654    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x6r6\" (UniqueName: \"kubernetes.io/projected/424be2c2-94d5-42d9-b434-d2bdaa809e85-kube-api-access-4x6r6\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.771721 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.769863    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424be2c2-94d5-42d9-b434-d2bdaa809e85-tls-certs\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.771721 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.770179    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-home\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.776898 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.776867    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x6r6\" (UniqueName: \"kubernetes.io/projected/424be2c2-94d5-42d9-b434-d2bdaa809e85-kube-api-access-4x6r6\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.777410 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.777378    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-model-cache\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.780047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.780024    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-tmp-dir\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.794478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.794445    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:03:53.804588 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.803420    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-dshm\") pod \"distinct-w6-0ed729-kserve-69c776cbb4-q687k\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:53.934990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:53.934694    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:54.380469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:54.380344    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"]
Nov 03 20:03:54.415975 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:03:54.415927    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod424be2c2_94d5_42d9_b434_d2bdaa809e85.slice/crio-2a0b1f07ad1c438435b849b0d515039d8e6649d756bc3ca132e5bbeb4af80ba5 WatchSource:0}: Error finding container 2a0b1f07ad1c438435b849b0d515039d8e6649d756bc3ca132e5bbeb4af80ba5: Status 404 returned error can't find the container with id 2a0b1f07ad1c438435b849b0d515039d8e6649d756bc3ca132e5bbeb4af80ba5
Nov 03 20:03:54.583551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:54.561036    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:03:54.583551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:54.563623    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:03:54.583551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:54.563669    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 20:03:54.670611 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:54.670567    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" event={"ID":"424be2c2-94d5-42d9-b434-d2bdaa809e85","Type":"ContainerStarted","Data":"2a0b1f07ad1c438435b849b0d515039d8e6649d756bc3ca132e5bbeb4af80ba5"}
Nov 03 20:03:54.802473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:54.801360    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:03:54.802473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:54.801417    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:03:55.548460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:55.547978    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:03:55.548460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:55.548043    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 20:03:55.700174 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:55.700134    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" event={"ID":"424be2c2-94d5-42d9-b434-d2bdaa809e85","Type":"ContainerStarted","Data":"6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462"}
Nov 03 20:03:55.742795 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:55.742718    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podStartSLOduration=2.742692214 podStartE2EDuration="2.742692214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:55.721026034 +0000 UTC m=+1418.233847825" watchObservedRunningTime="2026-11-03 20:03:55.742692214 +0000 UTC m=+1418.255513997"
Nov 03 20:03:55.795912 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:55.795875    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:03:55.796161 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:55.796129    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:03:55.945461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:55.944546    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:03:56.052576 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:56.052525    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:56.233202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:56.233101    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:56.547611 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:56.547519    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:03:56.547860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:56.547832    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 20:03:56.799513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:56.797853    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:03:56.799513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:56.797916    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:03:57.250460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:57.250347    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:57.264461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:57.255266    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:03:57.494629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:57.494587    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:03:57.547114 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:57.547009    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:03:57.550777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:57.550519    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 20:03:57.854545 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:57.843693    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:03:57.855134 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:57.855098    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:03:58.550559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:58.550072    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:03:58.550559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:58.550134    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 20:03:58.814959 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:58.814875    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:03:58.815270 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:58.815223    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:03:59.290541 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:03:59.290486    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:03:59.548512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:59.547538    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:03:59.548512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:59.547609    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 20:03:59.795911 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:59.795879    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:03:59.796298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:03:59.796251    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:04:00.547900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:00.547859    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:04:00.548485 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:00.548419    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 20:04:00.796463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:00.796027    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:00.796463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:00.796092    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:04:01.551931 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:01.551895    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:04:01.552707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:01.552657    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 20:04:01.814514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:01.812044    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:01.814514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:01.812123    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:04:02.551234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:02.551197    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 20:04:02.551473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:02.551260    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 20:04:02.794553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:02.794510    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:02.795221 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:02.795187    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 20:04:03.935580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:03.935546    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:04:03.954412 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:03.954380    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:04:03.970907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:03.970879    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:04:03.971301 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:03.971272    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:04:03.972201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:03.972174    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:04:04.335096 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:04.335009    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:04:04.693217 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:04.693186    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:04:04.693486 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:04.693468    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:04:04.694500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:04.694482    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:04:06.877207 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.877172    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"]
Nov 03 20:04:06.883385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.883359    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:06.892209 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.892179    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"openshift-service-ca.crt\""
Nov 03 20:04:06.892353 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.892322    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"default-dockercfg-c5bvs\""
Nov 03 20:04:06.892486 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.892468    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"kube-root-ca.crt\""
Nov 03 20:04:06.892590 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.892573    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"distinct-2-w0-120fe3-kserve-self-signed-certs\""
Nov 03 20:04:06.902018 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.901972    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"]
Nov 03 20:04:06.971531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.971495    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-tmp-dir\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:06.971757 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.971583    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpv7f\" (UniqueName: \"kubernetes.io/projected/447f04b7-254a-4df8-9581-db5e98d4b144-kube-api-access-qpv7f\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:06.971757 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.971703    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-model-cache\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:06.971904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.971756    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-dshm\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:06.971904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.971860    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447f04b7-254a-4df8-9581-db5e98d4b144-tls-certs\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:06.972015 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:06.971938    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-home\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.074152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.073464    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-home\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.074152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.073584    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-tmp-dir\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.074152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.073658    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpv7f\" (UniqueName: \"kubernetes.io/projected/447f04b7-254a-4df8-9581-db5e98d4b144-kube-api-access-qpv7f\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.074152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.073696    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-model-cache\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.074152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.073737    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-dshm\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.074152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.073785    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447f04b7-254a-4df8-9581-db5e98d4b144-tls-certs\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.076810 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.076004    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-home\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.076810 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.076251    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-tmp-dir\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.076810 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.076516    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-model-cache\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.080413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.077058    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"]
Nov 03 20:04:07.101565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.100729    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.125948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.125906    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpv7f\" (UniqueName: \"kubernetes.io/projected/447f04b7-254a-4df8-9581-db5e98d4b144-kube-api-access-qpv7f\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.201195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.200717    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.201195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.200768    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.201195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.200797    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xtbh\" (UniqueName: \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-kube-api-access-9xtbh\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.201195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.200866    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.201195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.200895    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.201195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.200936    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.201195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.200990    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.201195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.201069    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.201195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.201196    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.201814 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.201382    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"]
Nov 03 20:04:07.202738 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.202418    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447f04b7-254a-4df8-9581-db5e98d4b144-tls-certs\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.202738 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.202680    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-dshm\") pod \"distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") " pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.209285 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.209260    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"]
Nov 03 20:04:07.215985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.215960    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:07.218272 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.218254    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"]
Nov 03 20:04:07.218400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.218346    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.306846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.306804    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.307081 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.307063    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.307229 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.307214    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.307348 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.307332    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.307474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.307459    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d51280-043d-411f-aea0-4af2eb678c66-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.307600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.307586    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4vlp\" (UniqueName: \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-kube-api-access-h4vlp\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.307737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.307723    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.307838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.307825    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.307944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.307930    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtbh\" (UniqueName: \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-kube-api-access-9xtbh\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.308042 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.308027    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d51280-043d-411f-aea0-4af2eb678c66-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.308190 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.308175    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.308291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.308278    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.308834 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.308802    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.308993 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.308968    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.309108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.309091    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.309204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.309192    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.309338 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.309325    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.309514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.309500    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.310070 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.310033    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.310517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.310496    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.313782 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.313755    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.314746 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.314727    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.318427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.318338    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.318792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.318757    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.322919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.322895    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.339164 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.339135    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"]
Nov 03 20:04:07.354045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.353986    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.359977 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.359945    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.365664 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.364018    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"]
Nov 03 20:04:07.398365 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.398294    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtbh\" (UniqueName: \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-kube-api-access-9xtbh\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.410653 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.410540    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.410653 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.410583    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69p7\" (UniqueName: \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-kube-api-access-r69p7\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.410653 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.410629    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d51280-043d-411f-aea0-4af2eb678c66-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.411397 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411371    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d51280-043d-411f-aea0-4af2eb678c66-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.411544 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411524    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.411612 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411583    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.411751 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411670    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.411751 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411712    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.411751 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411734    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.411932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411780    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.411932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411800    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.411932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411816    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.411932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411892    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.411932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411911    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.411932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411926    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.412252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411964    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.412252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.411984    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d51280-043d-411f-aea0-4af2eb678c66-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.412252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.412000    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.412252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.412022    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4vlp\" (UniqueName: \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-kube-api-access-h4vlp\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.412252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.412030    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.412540 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.412290    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.412540 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.412315    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.412815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.412790    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.415106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.415085    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d51280-043d-411f-aea0-4af2eb678c66-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.415588 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.415546    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.419983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.419964    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.420560 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.420543    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4vlp\" (UniqueName: \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-kube-api-access-h4vlp\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.491974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.491944    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:07.513399 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.513359    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.513556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.513406    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.513556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.513483    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.513556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.513531    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.513556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.513554    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69p7\" (UniqueName: \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-kube-api-access-r69p7\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.513766 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.513605    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.513766 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.513653    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.513766 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.513749    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.513929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.513780    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.513929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.513787    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.514131 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.514066    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.514131 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.514094    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.514131 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.514124    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.514519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.514495    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.516261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.516239    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.516375 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.516357    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.521870 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.521852    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.522834 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.522813    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69p7\" (UniqueName: \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-kube-api-access-r69p7\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.527615 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.527589    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"]
Nov 03 20:04:07.528895 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:04:07.528867    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod447f04b7_254a_4df8_9581_db5e98d4b144.slice/crio-5abeb91466409bd8cd1509882a9a5afdd632d2819c1932072959186ec409358e WatchSource:0}: Error finding container 5abeb91466409bd8cd1509882a9a5afdd632d2819c1932072959186ec409358e: Status 404 returned error can't find the container with id 5abeb91466409bd8cd1509882a9a5afdd632d2819c1932072959186ec409358e
Nov 03 20:04:07.579077 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.572964    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:07.683267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.681285    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:07.811082 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.811045    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"]
Nov 03 20:04:07.825464 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:04:07.825396    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecdfd4b_64e1_4d0a_9114_a78870ae064a.slice/crio-c4b22f091dc7b719ba2245eeb01541e7ea22ff9cc2b478f74f9bf18c70561fed WatchSource:0}: Error finding container c4b22f091dc7b719ba2245eeb01541e7ea22ff9cc2b478f74f9bf18c70561fed: Status 404 returned error can't find the container with id c4b22f091dc7b719ba2245eeb01541e7ea22ff9cc2b478f74f9bf18c70561fed
Nov 03 20:04:07.829104 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.829064    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:07.829222 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.829166    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:07.829222 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.829213    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:07.891803 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.891774    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"]
Nov 03 20:04:07.901872 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:04:07.901842    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d51280_043d_411f_aea0_4af2eb678c66.slice/crio-de0af7d92dba29cf3987ed2e8bd88ac26e337f0afa25baea70ad0e7279090a5f WatchSource:0}: Error finding container de0af7d92dba29cf3987ed2e8bd88ac26e337f0afa25baea70ad0e7279090a5f: Status 404 returned error can't find the container with id de0af7d92dba29cf3987ed2e8bd88ac26e337f0afa25baea70ad0e7279090a5f
Nov 03 20:04:07.904852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.904820    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:07.904942 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.904897    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:07.904942 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:07.904928    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:08.006779 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:04:08.006734    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1e0711_ae1b_4ab2_9be7_f391f55e5a8d.slice/crio-2e6274eb982790203ebb0abcf83b892171494412fd0195e2a15a002e7fa1327e WatchSource:0}: Error finding container 2e6274eb982790203ebb0abcf83b892171494412fd0195e2a15a002e7fa1327e: Status 404 returned error can't find the container with id 2e6274eb982790203ebb0abcf83b892171494412fd0195e2a15a002e7fa1327e
Nov 03 20:04:08.007637 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.007609    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"]
Nov 03 20:04:08.010875 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.010839    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:08.010973 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.010922    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:08.010973 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.010963    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:08.372415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.372331    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" event={"ID":"447f04b7-254a-4df8-9581-db5e98d4b144","Type":"ContainerStarted","Data":"df22a4b39d0e96783baf1030381b0ac277bc4418e7a2ce96d65d05a7b04b6d54"}
Nov 03 20:04:08.372753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.372722    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" event={"ID":"447f04b7-254a-4df8-9581-db5e98d4b144","Type":"ContainerStarted","Data":"5abeb91466409bd8cd1509882a9a5afdd632d2819c1932072959186ec409358e"}
Nov 03 20:04:08.375062 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.374537    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" event={"ID":"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d","Type":"ContainerStarted","Data":"3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8"}
Nov 03 20:04:08.375258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.375236    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" event={"ID":"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d","Type":"ContainerStarted","Data":"2e6274eb982790203ebb0abcf83b892171494412fd0195e2a15a002e7fa1327e"}
Nov 03 20:04:08.376548 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.376527    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" event={"ID":"e0d51280-043d-411f-aea0-4af2eb678c66","Type":"ContainerStarted","Data":"5157ed0588b6bb223360454f3c1c24d8500d866bd4f0a2193dc0fff35fe66f95"}
Nov 03 20:04:08.376645 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.376571    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" event={"ID":"e0d51280-043d-411f-aea0-4af2eb678c66","Type":"ContainerStarted","Data":"de0af7d92dba29cf3987ed2e8bd88ac26e337f0afa25baea70ad0e7279090a5f"}
Nov 03 20:04:08.378358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.378335    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" event={"ID":"7ecdfd4b-64e1-4d0a-9114-a78870ae064a","Type":"ContainerStarted","Data":"04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5"}
Nov 03 20:04:08.378456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.378389    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" event={"ID":"7ecdfd4b-64e1-4d0a-9114-a78870ae064a","Type":"ContainerStarted","Data":"c4b22f091dc7b719ba2245eeb01541e7ea22ff9cc2b478f74f9bf18c70561fed"}
Nov 03 20:04:08.392426 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.392370    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podStartSLOduration=2.392355961 podStartE2EDuration="2.392355961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:08.39169913 +0000 UTC m=+1430.904520916" watchObservedRunningTime="2026-11-03 20:04:08.392355961 +0000 UTC m=+1430.905177744"
Nov 03 20:04:08.427848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.427758    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" podStartSLOduration=1.427736876 podStartE2EDuration="1.427736876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:08.417094962 +0000 UTC m=+1430.929916748" watchObservedRunningTime="2026-11-03 20:04:08.427736876 +0000 UTC m=+1430.940558660"
Nov 03 20:04:08.445461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.441478    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" podStartSLOduration=1.441458031 podStartE2EDuration="1.441458031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:08.436856581 +0000 UTC m=+1430.949678452" watchObservedRunningTime="2026-11-03 20:04:08.441458031 +0000 UTC m=+1430.954279808"
Nov 03 20:04:08.478458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.474581    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" podStartSLOduration=1.474541315 podStartE2EDuration="1.474541315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:08.473903439 +0000 UTC m=+1430.986725222" watchObservedRunningTime="2026-11-03 20:04:08.474541315 +0000 UTC m=+1430.987363097"
Nov 03 20:04:08.495462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.494788    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:08.504459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.503563    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:08.504459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.503636    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:04:08.580483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.579347    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:08.580483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.579522    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:04:08.580483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.579582    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:04:08.687362 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:08.687332    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:09.232084 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:09.232043    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:09.496956 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:09.496843    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:09.498580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:09.498549    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:04:09.578679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:09.578646    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:04:09.578854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:09.578696    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:04:09.689568 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:09.689535    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:09.689829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:09.689798    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:04:10.513957 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:10.513923    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:10.514943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:10.514557    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:04:10.578651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:10.578618    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:04:10.578910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:10.578886    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:04:10.687319 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:10.687283    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:10.687646 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:10.687614    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:04:11.517462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:11.509977    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:11.517462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:11.510038    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:04:11.636474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:11.591000    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:04:11.636474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:11.591099    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:04:11.746460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:11.742933    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:11.746460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:11.743003    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:04:12.535509 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:12.535470    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:12.537096 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:12.537064    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:04:12.576238 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:12.576186    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:04:12.576591 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:12.576559    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:04:12.681894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:12.681854    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:12.682177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:12.682147    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:04:13.493882 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:13.493841    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.494177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:13.494147    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:04:13.574289 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:13.573821    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.574289 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:13.573883    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:04:13.787472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:13.787364    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.791546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:13.787715    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:04:14.496458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:14.495855    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.496458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:14.495919    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:04:14.574680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:14.574642    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.575112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:14.574711    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:04:14.686459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:14.685827    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.686459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:14.685891    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:04:15.685819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:15.685783    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:15.686368 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:15.686337    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:04:16.002512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:16.001762    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:16.006458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:16.003295    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:16.006458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:16.004677    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:04:16.062379 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:16.061061    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:16.062792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:16.062627    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:16.063795 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:16.063745    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:04:16.729802 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:16.729766    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:16.731106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:16.731084    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:16.735154 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:16.735134    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:04:17.216856 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:17.216821    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:17.231775 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:17.231748    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:18.003140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.003102    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:18.340768 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.340677    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"]
Nov 03 20:04:18.346099 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.346073    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.358760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.358731    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"kube-root-ca.crt\""
Nov 03 20:04:18.359865 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.359831    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"openshift-service-ca.crt\""
Nov 03 20:04:18.360028 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.359980    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"default-dockercfg-5t6p5\""
Nov 03 20:04:18.360095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.360085    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"distinct-2-w5-ab1e46-kserve-self-signed-certs\""
Nov 03 20:04:18.364177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.364152    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"]
Nov 03 20:04:18.416613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.416582    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-home\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.416869 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.416845    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tmp-dir\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.417046 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.417013    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-model-cache\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.417190 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.417122    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-dshm\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.417190 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.417178    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tls-certs\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.417315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.417253    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6f8z\" (UniqueName: \"kubernetes.io/projected/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-kube-api-access-k6f8z\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.518357 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.518315    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-dshm\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.532673 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.532634    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tls-certs\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.537325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.534085    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6f8z\" (UniqueName: \"kubernetes.io/projected/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-kube-api-access-k6f8z\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.537325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.535738    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-home\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.537325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.535812    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tmp-dir\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.537325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.535921    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-model-cache\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.537325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.536404    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-model-cache\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.537325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.536730    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-home\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.537325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.536956    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tmp-dir\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.550704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.550670    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-dshm\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.574050 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.574012    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tls-certs\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.579469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.575044    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6f8z\" (UniqueName: \"kubernetes.io/projected/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-kube-api-access-k6f8z\") pod \"distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.674023 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.673961    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:18.985067 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:18.979578    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"]
Nov 03 20:04:19.002052 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:19.002010    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" event={"ID":"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34","Type":"ContainerStarted","Data":"b72d234c9eb2f436aefcddb45cf885babfb52a1b112f0e06cb3998a37a0ab1b9"}
Nov 03 20:04:20.029465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:20.027745    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" event={"ID":"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34","Type":"ContainerStarted","Data":"3b63cb68ca93db40ff17d9f93882377f6d3627e9a1bdb9e4f498c511ccb5276e"}
Nov 03 20:04:20.051637 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:20.051575    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podStartSLOduration=2.051553755 podStartE2EDuration="2.051553755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:20.048935349 +0000 UTC m=+1442.561757134" watchObservedRunningTime="2026-11-03 20:04:20.051553755 +0000 UTC m=+1442.564375539"
Nov 03 20:04:20.676555 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:20.676518    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:22.018325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.015489    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"]
Nov 03 20:04:22.090546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.063811    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"]
Nov 03 20:04:22.090546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.063937    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.201049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.200295    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.201049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.200386    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.201049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.200413    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.201049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.200491    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4vr\" (UniqueName: \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-kube-api-access-zl4vr\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.201049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.200567    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.201049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.200620    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.201049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.200686    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.201049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.200714    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.201049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.200763    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.330813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.329747    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.330813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.329799    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.330813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.329849    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.330813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.329938    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.330813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.330007    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.330813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.330033    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.330813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.330095    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4vr\" (UniqueName: \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-kube-api-access-zl4vr\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.330813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.330172    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.330813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.330241    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.416462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.384379    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.424502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.395083    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.424702 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.418605    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.424798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.418820    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.424910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.420314    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.426285 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.426248    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:22.483808 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.462498    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"]
Nov 03 20:04:22.498049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.497073    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:22.516207 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.515713    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.656141 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.656096    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.656481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.656457    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.656619 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.656602    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mr4\" (UniqueName: \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-kube-api-access-77mr4\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.656753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.656736    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.679481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.679424    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.683510 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.683462    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.683699 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.683683    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.683850 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.683836    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.684024 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.684011    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.727518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.661733    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.749559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.698769    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"]
Nov 03 20:04:22.749771 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.699820    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.783513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.783477    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.789787 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.789745    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.791284 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.791229    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.791466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.791426    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.791638 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.791620    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.793418 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.792359    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.793694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.793675    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.794283 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.794127    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.794578 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.794559    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.795073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.794944    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.795335 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.795316    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.795901 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.795882    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mr4\" (UniqueName: \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-kube-api-access-77mr4\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.796046 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.796030    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.799799 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.799777    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.802798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.801585    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.814894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.814861    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4vr\" (UniqueName: \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-kube-api-access-zl4vr\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:22.820445 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.820323    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.822137 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.822106    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.823175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.823140    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.824290 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.824234    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mr4\" (UniqueName: \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-kube-api-access-77mr4\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:22.860053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:22.859735    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:23.002889 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.001669    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:23.266477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.266418    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"]
Nov 03 20:04:23.272286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.271732    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:23.272286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.271816    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:23.272286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.271859    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:23.388753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.388535    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"]
Nov 03 20:04:23.395088 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.394396    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:23.395088 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.394496    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:23.395088 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.394538    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:23.454522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.454487    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" event={"ID":"436453e2-622e-4a77-aa2e-20c7dc39f6ad","Type":"ContainerStarted","Data":"70e1d7ad4c61c9e5f8962123811a5774802c79bd159810d907b6e70a89be6e4f"}
Nov 03 20:04:23.457887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.456711    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" event={"ID":"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c","Type":"ContainerStarted","Data":"0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738"}
Nov 03 20:04:23.457887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.456773    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" event={"ID":"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c","Type":"ContainerStarted","Data":"00e213e53304ef11729aaa7394ea40612315c134a4e55fe489ccccb50f0bd8a6"}
Nov 03 20:04:23.481774 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.479591    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" podStartSLOduration=2.479571207 podStartE2EDuration="2.479571207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:23.478911647 +0000 UTC m=+1445.991733433" watchObservedRunningTime="2026-11-03 20:04:23.479571207 +0000 UTC m=+1445.992392991"
Nov 03 20:04:23.861353 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:23.861276    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:24.465671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:24.465634    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" event={"ID":"436453e2-622e-4a77-aa2e-20c7dc39f6ad","Type":"ContainerStarted","Data":"1de9ac353fb84b54c4a6a14127f0369cb7532a183f27b4d8dce93d666a9bb497"}
Nov 03 20:04:24.516459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:24.502071    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" podStartSLOduration=2.502049999 podStartE2EDuration="2.502049999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:24.48611484 +0000 UTC m=+1446.998936622" watchObservedRunningTime="2026-11-03 20:04:24.502049999 +0000 UTC m=+1447.014871783"
Nov 03 20:04:24.863735 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:24.863648    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:24.863735 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:24.863700    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:04:25.007000 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:25.006952    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:25.007416 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:25.007395    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:25.007613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:25.007590    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:04:25.861594 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:25.861563    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:25.861984 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:25.861612    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:04:26.013085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:26.013049    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:26.013364 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:26.013322    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:04:26.861570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:26.861481    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:26.861570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:26.861534    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:04:27.005884 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:27.005846    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:27.006177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:27.006149    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:04:27.863040 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:27.862999    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:27.863614 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:27.863067    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:04:28.003459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:28.002857    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:28.003459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:28.002918    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:04:28.674690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:28.674660    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:28.776005 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:28.775971    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:28.869170 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:28.869123    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:28.869707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:28.869190    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:04:28.873195 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:04:28.873163    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:04:29.002965 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:29.002870    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:29.003231 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:29.003200    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:04:29.576161 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:29.576136    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:29.861454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:29.861340    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:29.861454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:29.861394    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:04:30.007462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:30.005847    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:30.007462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:30.005906    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:04:31.003474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:31.002723    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:31.003474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:31.002786    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:04:31.056052 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:31.056018    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:31.058139 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:31.058114    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:31.059893 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:31.059866    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:04:32.009687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:32.009647    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:32.010158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:32.010132    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:32.011415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:32.011388    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:04:35.823944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:35.823912    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"]
Nov 03 20:04:35.831719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:35.831690    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:35.841889 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:35.841864    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0ed729\"/\"distinct-2-w6-0ed729-kserve-self-signed-certs\""
Nov 03 20:04:35.860809 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:35.860786    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"]
Nov 03 20:04:35.916650 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:35.916616    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-home\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:35.916650 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:35.916648    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-tmp-dir\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:35.916865 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:35.916675    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdr76\" (UniqueName: \"kubernetes.io/projected/5d1e6564-0a1c-40d3-b800-665251906b46-kube-api-access-cdr76\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:35.916865 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:35.916820    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-dshm\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:35.916942 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:35.916867    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-model-cache\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:35.917095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:35.917072    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1e6564-0a1c-40d3-b800-665251906b46-tls-certs\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.018388 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.018341    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1e6564-0a1c-40d3-b800-665251906b46-tls-certs\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.019260 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.019236    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-home\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.019453 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.019419    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-tmp-dir\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.019624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.019605    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdr76\" (UniqueName: \"kubernetes.io/projected/5d1e6564-0a1c-40d3-b800-665251906b46-kube-api-access-cdr76\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.034027 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.034000    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-dshm\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.034688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.034665    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-model-cache\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.035553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.031636    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-home\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.035729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.035179    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-model-cache\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.035867 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.031897    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-tmp-dir\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.039077 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.039055    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1e6564-0a1c-40d3-b800-665251906b46-tls-certs\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.041847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.041826    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-dshm\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.056522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.056405    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdr76\" (UniqueName: \"kubernetes.io/projected/5d1e6564-0a1c-40d3-b800-665251906b46-kube-api-access-cdr76\") pod \"distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") " pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.156017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.155981    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:36.477535 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:04:36.477500    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d1e6564_0a1c_40d3_b800_665251906b46.slice/crio-cfde90ee0bb6b551321c189f5be6f6f0228b71d26b725c9a34f3ce00f7192c7d WatchSource:0}: Error finding container cfde90ee0bb6b551321c189f5be6f6f0228b71d26b725c9a34f3ce00f7192c7d: Status 404 returned error can't find the container with id cfde90ee0bb6b551321c189f5be6f6f0228b71d26b725c9a34f3ce00f7192c7d
Nov 03 20:04:36.478780 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.478753    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"]
Nov 03 20:04:36.617948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.617908    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" event={"ID":"5d1e6564-0a1c-40d3-b800-665251906b46","Type":"ContainerStarted","Data":"0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef"}
Nov 03 20:04:36.618151 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:36.617964    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" event={"ID":"5d1e6564-0a1c-40d3-b800-665251906b46","Type":"ContainerStarted","Data":"cfde90ee0bb6b551321c189f5be6f6f0228b71d26b725c9a34f3ce00f7192c7d"}
Nov 03 20:04:37.156778 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.156742    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:37.327536 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.327477    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podStartSLOduration=2.327427438 podStartE2EDuration="2.327427438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:36.638652473 +0000 UTC m=+1459.151474254" watchObservedRunningTime="2026-11-03 20:04:37.327427438 +0000 UTC m=+1459.840249221"
Nov 03 20:04:37.329029 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.329001    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"]
Nov 03 20:04:37.334494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.334466    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.351926 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.351895    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"]
Nov 03 20:04:37.363756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.363731    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.363939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.363766    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.363939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.363795    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.364071 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.364017    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.364071 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.364056    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.364154 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.364134    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29cfda3e-9584-49b9-a9b9-4824e6efa782-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.364201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.364185    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.364290 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.364278    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.364328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.364316    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqr65\" (UniqueName: \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-kube-api-access-jqr65\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.465452 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.465361    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.465452 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.465428    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.465629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.465489    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqr65\" (UniqueName: \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-kube-api-access-jqr65\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.465629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.465546    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.465629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.465570    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.465629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.465607    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.465854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.465721    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.465854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.465746    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.465854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.465802    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29cfda3e-9584-49b9-a9b9-4824e6efa782-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.466037 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.465860    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.466171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.466053    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.466256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.466220    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.466256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.465907    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.467093 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.466853    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29cfda3e-9584-49b9-a9b9-4824e6efa782-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.468636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.468615    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.468636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.468620    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.480856 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.480833    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.481646 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.481621    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"]
Nov 03 20:04:37.483196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.483172    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqr65\" (UniqueName: \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-kube-api-access-jqr65\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.489020 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.489003    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.499302 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.499278    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"]
Nov 03 20:04:37.567075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.567038    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e39420c-d71a-475a-8f31-42ab6a7ea412-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.567285 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.567092    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.567367 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.567278    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.567367 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.567312    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.567367 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.567343    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.567602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.567374    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.567602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.567491    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stl5x\" (UniqueName: \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-kube-api-access-stl5x\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.567602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.567522    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.567755 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.567621    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.656561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.656525    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:37.669557 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.669521    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stl5x\" (UniqueName: \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-kube-api-access-stl5x\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.669692 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.669597    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.669770 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.669691    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.669826 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.669797    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e39420c-d71a-475a-8f31-42ab6a7ea412-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.669879 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.669828    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.669984 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.669967    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.670042 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.669999    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.670042 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.670024    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.670150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.670056    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.670605 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.670579    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.670756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.670734    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.671050 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.671030    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.671295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.671276    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.671660 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.671636    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e39420c-d71a-475a-8f31-42ab6a7ea412-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.672987 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.672963    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.673283 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.673265    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.677708 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.677685    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stl5x\" (UniqueName: \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-kube-api-access-stl5x\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.678669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.678651    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.846772 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.846738    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:37.969809 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.969719    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"]
Nov 03 20:04:37.973941 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:04:37.973912    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cfda3e_9584_49b9_a9b9_4824e6efa782.slice/crio-aa2918ca1312c283b096dcc789c85713a787ee8bf5a22f4bc4a1cbcf0d925ea0 WatchSource:0}: Error finding container aa2918ca1312c283b096dcc789c85713a787ee8bf5a22f4bc4a1cbcf0d925ea0: Status 404 returned error can't find the container with id aa2918ca1312c283b096dcc789c85713a787ee8bf5a22f4bc4a1cbcf0d925ea0
Nov 03 20:04:37.978406 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.978375    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:37.978522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.978455    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:37.978522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:37.978486    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:38.170223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.170195    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"]
Nov 03 20:04:38.175858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.175820    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:38.175988 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.175895    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:38.175988 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.175939    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:38.647758 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.647716    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" event={"ID":"0e39420c-d71a-475a-8f31-42ab6a7ea412","Type":"ContainerStarted","Data":"8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd"}
Nov 03 20:04:38.647973 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.647788    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" event={"ID":"0e39420c-d71a-475a-8f31-42ab6a7ea412","Type":"ContainerStarted","Data":"5f99eff7cf2968361297c258f566afdd17d294988087f91d6a3da6b89ace820f"}
Nov 03 20:04:38.651481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.651447    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" event={"ID":"29cfda3e-9584-49b9-a9b9-4824e6efa782","Type":"ContainerStarted","Data":"94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6"}
Nov 03 20:04:38.651650 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.651497    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" event={"ID":"29cfda3e-9584-49b9-a9b9-4824e6efa782","Type":"ContainerStarted","Data":"aa2918ca1312c283b096dcc789c85713a787ee8bf5a22f4bc4a1cbcf0d925ea0"}
Nov 03 20:04:38.657788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.657762    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:38.674740 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.674682    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" podStartSLOduration=1.674663875 podStartE2EDuration="1.674663875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:38.670345585 +0000 UTC m=+1461.183167366" watchObservedRunningTime="2026-11-03 20:04:38.674663875 +0000 UTC m=+1461.187485658"
Nov 03 20:04:38.705458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.696261    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" podStartSLOduration=1.696242099 podStartE2EDuration="1.696242099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:38.693782504 +0000 UTC m=+1461.206604288" watchObservedRunningTime="2026-11-03 20:04:38.696242099 +0000 UTC m=+1461.209063882"
Nov 03 20:04:38.849559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:38.849497    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:39.658722 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:39.658692    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:04:39.659120 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:39.658744    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:04:39.915583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:39.891586    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:04:39.915583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:39.891646    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:04:40.658091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:40.658050    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:04:40.658386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:40.658133    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:04:40.856474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:40.855872    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:04:40.856474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:40.855927    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:04:41.663605 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:41.663564    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:04:41.663834 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:41.663656    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:04:41.854458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:41.849901    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:04:41.854458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:41.849964    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:04:42.658066 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:42.657949    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:04:42.658066 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:42.658020    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:04:42.853346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:42.852874    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:04:42.853346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:42.852949    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:04:43.657811 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:43.657767    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:04:43.658003 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:43.657829    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:04:43.860962 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:43.860923    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:04:43.861573 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:43.861536    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:04:44.657041 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:44.656999    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:04:44.657281 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:44.657087    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:04:44.852666 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:44.852627    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:04:44.852951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:44.852924    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:04:45.963892 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:45.963855    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:45.967467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:45.965833    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:45.972455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:45.971858    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:04:46.148878 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:46.148831    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:46.149838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:46.149812    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:46.152032 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:46.152011    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:04:46.157563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:46.156668    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:46.180358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:46.180330    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:46.838291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:46.838263    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:50.522936 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:50.522895    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:50.571705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:50.571659    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:50.840515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:50.837553    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:04:51.004868 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:51.003630    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:51.080801 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:51.079570    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:51.334461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:51.330532    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:51.557460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:51.532615    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:04:51.646522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:51.591989    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:04:51.646522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:51.591665    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:51.646522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:51.593706    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:51.852462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:51.851655    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:51.978525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:51.972708    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:52.060499 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:52.014507    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:52.131462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:52.076546    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:52.342765 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:52.342660    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:04:52.370027 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:52.343930    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:04:52.628230 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:52.627466    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:04:52.628230 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:52.627571    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:52.844704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:52.844658    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"]
Nov 03 20:04:52.879824 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:52.879698    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:04:52.918460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:52.904242    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:52.979478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:52.975829    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"]
Nov 03 20:04:53.024720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.023949    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.024720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.024026    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeded9b3-ac9b-4390-81db-6001a219415f-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.024720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.024161    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2pcv\" (UniqueName: \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-kube-api-access-b2pcv\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.024720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.024223    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.024720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.024253    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.024720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.024282    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.024720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.024408    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.024720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.024482    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.024720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.024507    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeded9b3-ac9b-4390-81db-6001a219415f-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.071673 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.071628    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:04:53.100469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.072238    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:04:53.128345 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.128298    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.129331 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.129307    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.129983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.129930    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeded9b3-ac9b-4390-81db-6001a219415f-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.130610 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.129874    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.130798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.130779    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.131312 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.131291    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeded9b3-ac9b-4390-81db-6001a219415f-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.132491 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.132395    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2pcv\" (UniqueName: \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-kube-api-access-b2pcv\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.132969 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.132947    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.133452 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.133410    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.133880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.133862    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.143136 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.132216    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeded9b3-ac9b-4390-81db-6001a219415f-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.146537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.133803    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.146737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.133344    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.146845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.134241    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.169066 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.169031    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.205284 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.201897    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl" podUID="adbf5e48-bc06-4df3-aa7d-fe2731c194f8" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.205284 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.203221    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2pcv\" (UniqueName: \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-kube-api-access-b2pcv\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.205284 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.203806    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.315498 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.305615    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeded9b3-ac9b-4390-81db-6001a219415f-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.344630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.344566    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.443455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.405302    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:53.730526 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.717018    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:04:53.730526 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.721060    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.811596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:53.790228    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:04:54.109521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:54.098697    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:54.183274 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:54.181716    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:54.183274 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:54.183225    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:04:54.748504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:54.743168    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:04:54.978461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:54.977912    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:55.149536 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:55.089885    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:55.217386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:55.203889    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:04:55.253735 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:55.227129    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:55.262958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:55.262255    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:04:55.531601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:55.531091    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:55.531601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:55.531189    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:55.531601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:55.531232    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:04:55.598243 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:55.597335    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:04:55.675654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:55.675625    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"]
Nov 03 20:04:55.753577 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:55.743798    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:55.871553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:55.870413    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:04:56.217015 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:56.216969    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:56.269014 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:56.268970    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:56.279169 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:56.279140    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:04:56.298279 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:56.298253    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:04:56.323466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:56.321523    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" event={"ID":"aeded9b3-ac9b-4390-81db-6001a219415f","Type":"ContainerStarted","Data":"5275867c5aa874046fea0fd03c632a3ccd55ba79cb5567504d3cf5c698dfe674"}
Nov 03 20:04:56.323466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:56.321597    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" event={"ID":"aeded9b3-ac9b-4390-81db-6001a219415f","Type":"ContainerStarted","Data":"7539b947d77973df3c6167a7c1e08b720ccac729a5676b34d6bb1a9e67a321b3"}
Nov 03 20:04:56.323466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:56.322710    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:04:56.417298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:56.417207    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:04:56.433574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:56.431555    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:04:56.433574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:56.431614    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:04:57.414346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:57.413868    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:04:57.414346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:57.413934    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:04:58.415464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:58.405857    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:04:58.415464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:58.405947    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:04:58.826604 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:04:58.826514    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:04:59.406863 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:59.406822    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:04:59.407178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:04:59.407146    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:05:00.410459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:00.408887    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:05:00.410459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:00.408953    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:05:01.416187 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:01.416148    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:05:01.416736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:01.416215    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:05:02.406859 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:02.406820    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:05:02.407198 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:02.407165    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:05:03.405975 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:03.405933    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:05:03.406413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:03.406004    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:05:04.411188 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:04.411157    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:05:04.411633 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:04.411482    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:05:04.412259 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:04.412201    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:05:04.432669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:04.432589    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" podStartSLOduration=12.432573667 podStartE2EDuration="12.432573667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:56.456610878 +0000 UTC m=+1478.969432679" watchObservedRunningTime="2026-11-03 20:05:04.432573667 +0000 UTC m=+1486.945395451"
Nov 03 20:05:07.772804 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.772771    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"]
Nov 03 20:05:07.776858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.776835    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.794135 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.794109    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"]
Nov 03 20:05:07.876532 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.876488    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73112745-0b21-4933-856a-0830555ca2a3-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.876708 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.876547    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.876755 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.876713    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73112745-0b21-4933-856a-0830555ca2a3-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.876755 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.876740    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wjcf\" (UniqueName: \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-kube-api-access-7wjcf\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.876865 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.876787    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.876865 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.876837    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.876972 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.876950    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.877025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.876995    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.877080 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.877046    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978099 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978059    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978099 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978107    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978367 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978233    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978404    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73112745-0b21-4933-856a-0830555ca2a3-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978485    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978591 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978537    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978591 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978544    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73112745-0b21-4933-856a-0830555ca2a3-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978591 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978574    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wjcf\" (UniqueName: \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-kube-api-access-7wjcf\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978744 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978593    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978744 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978651    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978744 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978682    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978924 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978879    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.978991 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.978961    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.979158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.979137    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73112745-0b21-4933-856a-0830555ca2a3-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.980955 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.980926    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.981211 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.981192    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73112745-0b21-4933-856a-0830555ca2a3-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.986981 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.986953    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:07.987126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:07.987106    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wjcf\" (UniqueName: \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-kube-api-access-7wjcf\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:08.105020 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:08.104933    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:08.403496 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:08.403468    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"]
Nov 03 20:05:08.403655 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:05:08.403636    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73112745_0b21_4933_856a_0830555ca2a3.slice/crio-5ee559d925100fdfc48b1f2fcf260f95fadb299a97cdfa2749d822f075895629 WatchSource:0}: Error finding container 5ee559d925100fdfc48b1f2fcf260f95fadb299a97cdfa2749d822f075895629: Status 404 returned error can't find the container with id 5ee559d925100fdfc48b1f2fcf260f95fadb299a97cdfa2749d822f075895629
Nov 03 20:05:08.406171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:08.406139    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:08.406292 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:08.406214    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:08.406292 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:08.406245    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:08.413684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:08.413659    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" event={"ID":"73112745-0b21-4933-856a-0830555ca2a3","Type":"ContainerStarted","Data":"5ee559d925100fdfc48b1f2fcf260f95fadb299a97cdfa2749d822f075895629"}
Nov 03 20:05:09.423797 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:09.423754    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" event={"ID":"73112745-0b21-4933-856a-0830555ca2a3","Type":"ContainerStarted","Data":"c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5"}
Nov 03 20:05:09.455030 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:09.454979    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" podStartSLOduration=2.454963846 podStartE2EDuration="2.454963846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:09.451865857 +0000 UTC m=+1491.964687643" watchObservedRunningTime="2026-11-03 20:05:09.454963846 +0000 UTC m=+1491.967785628"
Nov 03 20:05:10.107456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:10.106868    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:10.112603 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:10.112565    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:05:10.112786 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:10.112650    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" podUID="73112745-0b21-4933-856a-0830555ca2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:05:11.120465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:11.106228    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:05:11.142474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:11.141616    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" podUID="73112745-0b21-4933-856a-0830555ca2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:05:12.108804 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:12.108774    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:05:12.109578 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:12.109546    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" podUID="73112745-0b21-4933-856a-0830555ca2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:05:13.106858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:13.106822    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:05:13.107512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:13.107478    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" podUID="73112745-0b21-4933-856a-0830555ca2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:05:14.113608 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:14.113568    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:05:14.114504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:14.114465    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" podUID="73112745-0b21-4933-856a-0830555ca2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:05:15.105490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:15.105455    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:05:15.105682 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:15.105529    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" podUID="73112745-0b21-4933-856a-0830555ca2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:05:16.112227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:16.112185    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:16.112861 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:16.112800    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:16.114572 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:16.114550    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:05:22.835461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.834410    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"]
Nov 03 20:05:22.855799 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.854962    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:22.887714 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.887626    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"]
Nov 03 20:05:22.984460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.969366    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:22.984460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.969506    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:22.984460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.969557    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:22.984460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.969602    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:22.984460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.969662    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:22.984460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.969711    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7cx\" (UniqueName: \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-kube-api-access-xs7cx\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:22.984460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.969738    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:22.984460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.969783    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:22.984460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:22.969900    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.078678    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.078746    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7cx\" (UniqueName: \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-kube-api-access-xs7cx\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.078783    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.078827    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.078939    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.079043    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.079140    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.079188    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.079236    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.079738    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.080021    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.080914    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.084344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.081152    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.105813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.093995    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.205699 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.204018    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.206998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.206942    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.220463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.208079    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7cx\" (UniqueName: \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-kube-api-access-xs7cx\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.230467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.223088    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.504464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.501615    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:23.683411 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.659545    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.729109 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.729071    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.892946 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:23.892903    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:24.155985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:24.154741    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:24.355655 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:24.345531    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:05:24.578263 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:24.578158    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"]
Nov 03 20:05:24.891516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:24.747513    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:05:24.891516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:24.748195    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:05:24.895561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:24.895513    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:24.923014 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:24.896728    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:05:25.117687 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:05:25.117647    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d84a43_032e_4b91_a73e_16f6d083fbf7.slice/crio-0d15638201f88769fb16e95ac65211224d2ea6251db9ff1cd2f0d58c5ca32b6d WatchSource:0}: Error finding container 0d15638201f88769fb16e95ac65211224d2ea6251db9ff1cd2f0d58c5ca32b6d: Status 404 returned error can't find the container with id 0d15638201f88769fb16e95ac65211224d2ea6251db9ff1cd2f0d58c5ca32b6d
Nov 03 20:05:25.230777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:25.223676    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded"
Nov 03 20:05:25.230777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:25.223971    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:25.230777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:25.228665    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:05:25.370474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:25.351009    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:05:25.396460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:25.371206    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:05:25.499472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:25.493587    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:25.499472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:25.493691    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:25.636487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:25.571460    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:25.980796 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:25.980746    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:05:25.993823 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:25.993779    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:05:26.082199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:26.082145    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:26.322511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:26.288248    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:26.601679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:26.474719    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:05:26.735473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:26.587779    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:05:27.103523 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:27.073262    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:27.136636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:27.136591    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:05:27.161459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:27.155550    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:27.320519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:27.194548    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:27.547462 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:05:27.543480    5600 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.833s"
Nov 03 20:05:27.570459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:27.568741    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:05:27.592270 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:27.588532    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:27.592270 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:27.589220    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:27.798462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:27.798139    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:28.107461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:28.099650    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:05:28.107461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:28.099722    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:05:28.107461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:28.099741    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" event={"ID":"f2d84a43-032e-4b91-a73e-16f6d083fbf7","Type":"ContainerStarted","Data":"0d15638201f88769fb16e95ac65211224d2ea6251db9ff1cd2f0d58c5ca32b6d"}
Nov 03 20:05:28.205464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:28.149554    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:28.256477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:28.249967    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:28.348403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:28.311047    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:28.440497 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:28.433653    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 03 20:05:28.599520 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:28.591003    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:28.640703 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:28.599729    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:05:28.715031 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:28.714018    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 03 20:05:29.098626 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.098578    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:05:29.099778 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.099737    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:29.186470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.178715    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:05:29.286757 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.286673    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:05:29.304969 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.304940    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:05:29.380306 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:05:29.379582    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:05:29.384003 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.382213    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:05:29.505354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.505327    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:29.593216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.593119    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:29.593216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.593185    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:05:29.630885 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.630844    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:29.701970 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.701911    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" podStartSLOduration=7.701894264 podStartE2EDuration="7.701894264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:29.594007062 +0000 UTC m=+1512.106828846" watchObservedRunningTime="2026-11-03 20:05:29.701894264 +0000 UTC m=+1512.214716046"
Nov 03 20:05:29.878624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.878534    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:05:29.879054 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:29.879035    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:05:30.426324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:30.426281    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" event={"ID":"f2d84a43-032e-4b91-a73e-16f6d083fbf7","Type":"ContainerStarted","Data":"5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913"}
Nov 03 20:05:30.512458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:30.506846    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:30.512458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:30.506907    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:05:31.503854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:31.503816    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:31.504509 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:31.504474    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:05:32.504462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:32.503477    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:32.504462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:32.503565    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:05:33.503462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:33.502866    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:33.503462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:33.502936    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:05:34.502897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:34.502863    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:34.503546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:34.503513    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:05:35.503068 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:35.503027    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:35.503612 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:35.503086    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:05:36.998728 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:36.998694    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:36.999323 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:36.999297    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:37.000658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:37.000635    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:05:52.330415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.330379    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"]
Nov 03 20:05:52.363759 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.363721    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.365499 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.365466    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"]
Nov 03 20:05:52.448089 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.448054    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.448089 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.448093    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.448295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.448184    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftpnz\" (UniqueName: \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-kube-api-access-ftpnz\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.448332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.448304    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.448366 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.448337    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.448366 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.448358    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.448516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.448498    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.448560 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.448532    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.448597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.448564    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.511673 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.511637    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"]
Nov 03 20:05:52.516740 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.516712    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.534143 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.533994    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"]
Nov 03 20:05:52.549845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.549813    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550029 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.549854    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550029 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.549898    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftpnz\" (UniqueName: \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-kube-api-access-ftpnz\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550029 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.549976    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550029 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.549997    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550029 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.550024    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.550142    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.550177    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.550234    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.550664    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550771 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.550701    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550835 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.550812    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.550888 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.550840    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.551331 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.551253    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.553199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.553153    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.553584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.553568    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.560913 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.560869    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.561558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.561534    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftpnz\" (UniqueName: \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-kube-api-access-ftpnz\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.651298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.651267    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.651528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.651308    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.651528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.651327    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.651636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.651530    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.651636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.651605    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzcjl\" (UniqueName: \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-kube-api-access-hzcjl\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.651636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.651632    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.651767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.651745    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.651807 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.651772    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.651807 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.651791    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.685123 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.685093    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:52.755333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.753640    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.755333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.753743    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.755333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.753779    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.755333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.753839    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.755333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.753879    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.755333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.753905    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.755333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.754083    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.755333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.754206    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzcjl\" (UniqueName: \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-kube-api-access-hzcjl\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.755333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.754231    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.755333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.754778    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.755333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.754962    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.756078 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.755375    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.756252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.756166    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.756908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.756866    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.788146 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.759332    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.788146 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.786092    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.804458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.804373    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzcjl\" (UniqueName: \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-kube-api-access-hzcjl\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.812109 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.812046    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:52.841082 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:52.841037    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:53.175151 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:53.175121    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"]
Nov 03 20:05:53.401563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:53.396267    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:53.401563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:53.396374    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:53.401563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:53.396420    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:53.785767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:53.785721    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:53.786026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:53.786002    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:53.786176 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:53.786157    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:05:53.837201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:53.837166    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" event={"ID":"0ae75d0d-c70f-4206-9eeb-f02c9b723626","Type":"ContainerStarted","Data":"8dbca043c8dd232a0bd95ac5062dd34325614d8dee704dfee013b9cff080bf2e"}
Nov 03 20:05:53.837500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:53.837479    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"]
Nov 03 20:05:54.200918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:54.198556    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:54.200918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:54.199878    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:05:54.604157 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:54.601726    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:05:54.604157 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:54.602086    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:05:54.711760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:54.710535    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:54.925461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:54.918219    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" event={"ID":"71d7ec61-4c37-4cfc-80fe-102b094dfe0c","Type":"ContainerStarted","Data":"5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211"}
Nov 03 20:05:54.925461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:54.918291    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" event={"ID":"71d7ec61-4c37-4cfc-80fe-102b094dfe0c","Type":"ContainerStarted","Data":"80afbeaa772d689161c0878e8c8c45a35e3247914629c94dd3499ee7e5df07d3"}
Nov 03 20:05:54.953577 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:54.953529    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded"
Nov 03 20:05:55.013207 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.013161    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:55.089651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.089327    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" event={"ID":"0ae75d0d-c70f-4206-9eeb-f02c9b723626","Type":"ContainerStarted","Data":"00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263"}
Nov 03 20:05:55.091551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.091498    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podStartSLOduration=3.091480979 podStartE2EDuration="3.091480979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:54.966639547 +0000 UTC m=+1537.479461355" watchObservedRunningTime="2026-11-03 20:05:55.091480979 +0000 UTC m=+1537.604302762"
Nov 03 20:05:55.204029 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.203925    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:55.204621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.204248    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:05:55.252662 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.204677    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:05:55.252977 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.252955    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:05:55.266229 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.266151    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podStartSLOduration=3.266129122 podStartE2EDuration="3.266129122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:55.157823867 +0000 UTC m=+1537.670645652" watchObservedRunningTime="2026-11-03 20:05:55.266129122 +0000 UTC m=+1537.778950907"
Nov 03 20:05:55.683549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.658529    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:05:55.685467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.665817    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:55.735300 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.735261    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:55.743721 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.743686    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:05:55.747549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:55.747512    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:05:56.171522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.068730    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.266742 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.266705    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:05:56.270563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.270520    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:05:56.283096 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.271476    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.309464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.290806    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:05:56.940925 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.709505    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.945057 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.709550    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.955542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.754339    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.955766 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:05:56.866830    5600 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.247s"
Nov 03 20:05:56.956026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.873595    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:05:56.956180 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.956152    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:05:56.956342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.877567    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:05:56.956480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.956456    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:05:56.976462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:56.972702    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:05:57.042368 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.042337    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:05:57.042829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.042807    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:05:57.043015 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.043001    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:05:57.043154 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.043142    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:05:57.043283 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.043272    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:05:57.081890 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.081847    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:05:57.310518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.285543    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:57.310518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.286044    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:57.475397 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.475365    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:05:57.932561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.932513    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:05:57.989495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.986826    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:05:57.989495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.960453    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:05:57.989495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.987175    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:05:57.989495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.985845    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:05:57.989495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.987220    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:05:57.989495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:57.960092    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:05:58.149461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:58.124093    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:05:58.184965 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:58.184368    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:58.436139 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:58.436030    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:58.694966 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:58.689117    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:05:58.694966 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:58.689194    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:05:58.849103 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:58.849060    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 03 20:05:58.849968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:58.849851    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:05:58.849968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:58.849904    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:05:58.978236 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:58.976999    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:58.992405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:58.990541    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:58.992405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:58.990922    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:05:59.236521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:59.179381    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:59.236521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:59.210776    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:59.299532 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:59.299483    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:59.560347 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:05:59.560245    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:05:59.677460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:59.676732    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:59.693465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:59.686015    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:05:59.693465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:59.686070    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:05:59.844423 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:05:59.844329    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:06:00.033347 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.030569    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.042461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.030608    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.042461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.032773    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:06:00.057108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.033297    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:06:00.213575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.213523    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.351666 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.351598    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.353087 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.353052    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:06:00.382075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.373890    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:06:00.626720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.626622    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:00.652463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.626968    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.719499 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.719378    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:00.719499 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.719472    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:06:01.038536 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:00.966722    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:06:01.038536 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.034516    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:06:01.038536 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.035069    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:01.038536 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.035369    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:01.038536 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.035644    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:01.281661 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.203321    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:01.281998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.281962    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:01.282166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.218618    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:06:01.390522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.388830    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:06:01.495515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.495463    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:06:01.541761 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.541682    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:01.542117 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.542085    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:01.587475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.587217    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:01.588075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.587849    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:01.987927 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:01.987877    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:06:02.016852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.016797    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.090338    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.208250    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.151400    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.151251    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.154236    5600 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.208858    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.180710    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.208917    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.203575    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.208957    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.208187    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:02.211636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.208993    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" podUID="73112745-0b21-4933-856a-0830555ca2a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:02.225673 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.225637    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:02.225941 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.225911    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:02.250547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.248540    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:02.283583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.283532    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:02.451567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.451217    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:06:02.505107 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.505005    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:02.686843 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.686805    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:02.785291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.767216    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:06:02.961692 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.961655    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:06:02.961960 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:02.961926    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:06:03.004570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.004535    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:03.004827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.004790    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:03.116835 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.116733    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:03.117102 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.117073    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:03.213774 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.213733    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:03.215376 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.215345    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:03.248828 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.248790    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:06:03.285743 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.285685    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:03.458391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.458347    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:03.512461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.510789    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:06:03.711800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.711702    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:03.712053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.712017    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:06:03.844583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.843602    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:06:03.844583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.843666    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:06:03.944307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:03.943005    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:06:04.214567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.214510    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:06:04.230580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.230539    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:04.253906 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.253867    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:06:04.266785 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.266031    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:06:04.286841 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.286798    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:04.291002 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.290978    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:06:04.454576 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.454545    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:06:04.764526 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.744574    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:04.764526 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.744633    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:06:04.766589 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.766136    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:06:04.767687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.767669    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:06:04.858525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.855463    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:06:04.858525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:04.855539    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:06:05.699233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:05.699180    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:05.699886 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:05.699849    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:06:05.847462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:05.841857    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:06:05.847462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:05.841927    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:06:06.703458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:06.685870    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:06.703458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:06.685931    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:06:06.843487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:06.842929    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:06:06.843487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:06.843021    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:06:07.355479 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.353271    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"]
Nov 03 20:06:07.361825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.361171    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"]
Nov 03 20:06:07.361825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.361292    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.465801 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.465765    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.465951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.465826    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.465951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.465852    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.465951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.465907    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02b908f-874f-437d-98c9-99032b73a1f5-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.465951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.465934    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02b908f-874f-437d-98c9-99032b73a1f5-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.466101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.465980    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.466101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.466024    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.466101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.466052    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqf5\" (UniqueName: \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-kube-api-access-4bqf5\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.466101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.466078    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567206 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567171    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567379 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567269    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02b908f-874f-437d-98c9-99032b73a1f5-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567379 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567308    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02b908f-874f-437d-98c9-99032b73a1f5-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567479 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567379    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567482    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567543 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567521    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqf5\" (UniqueName: \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-kube-api-access-4bqf5\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567569    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567618 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567593    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567681 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567667    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567733 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567723    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567871 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567851    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.567971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.567957    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.568071 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.568050    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02b908f-874f-437d-98c9-99032b73a1f5-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.568385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.568358    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.570245 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.570214    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.570545 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.570527    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02b908f-874f-437d-98c9-99032b73a1f5-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.639508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.616165    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqf5\" (UniqueName: \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-kube-api-access-4bqf5\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.639764 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.629583    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.679301 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.679272    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:07.693463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.685523    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:06:07.693463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.686594    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:07.693463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.686636    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:06:07.843939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.843904    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:06:07.844585 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.844549    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:06:07.852494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:07.845258    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:06:08.160207 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.152566    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:06:08.160207 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.152661    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:06:08.160207 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.152710    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:06:08.160207 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.159335    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"]
Nov 03 20:06:08.712458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.712052    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:08.712458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.712104    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:06:08.716499 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.714624    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" event={"ID":"a02b908f-874f-437d-98c9-99032b73a1f5","Type":"ContainerStarted","Data":"0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a"}
Nov 03 20:06:08.716499 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.714688    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" event={"ID":"a02b908f-874f-437d-98c9-99032b73a1f5","Type":"ContainerStarted","Data":"90ca2996b4f5fb56dce21477c5319b5f875a5d4148e92f3c165134f864421888"}
Nov 03 20:06:08.801139 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.801053    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" podStartSLOduration=1.801033664 podStartE2EDuration="1.801033664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:08.795978641 +0000 UTC m=+1551.308800424" watchObservedRunningTime="2026-11-03 20:06:08.801033664 +0000 UTC m=+1551.313855447"
Nov 03 20:06:08.854599 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.854551    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:06:08.855171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.855132    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:06:08.879986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.879953    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11053-e2d371"]
Nov 03 20:06:08.898648 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.898616    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11053-e2d371"]
Nov 03 20:06:08.937847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.899958    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11053-e2d371"
Nov 03 20:06:08.966505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:08.963125    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4mfkw\""
Nov 03 20:06:09.031524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:09.031484    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk6sd\" (UniqueName: \"kubernetes.io/projected/b394a6ef-7d75-42ec-80f2-4c691cc71420-kube-api-access-wk6sd\") pod \"test-curl-11053-e2d371\" (UID: \"b394a6ef-7d75-42ec-80f2-4c691cc71420\") " pod="opendatahub/test-curl-11053-e2d371"
Nov 03 20:06:09.134361 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:09.134313    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6sd\" (UniqueName: \"kubernetes.io/projected/b394a6ef-7d75-42ec-80f2-4c691cc71420-kube-api-access-wk6sd\") pod \"test-curl-11053-e2d371\" (UID: \"b394a6ef-7d75-42ec-80f2-4c691cc71420\") " pod="opendatahub/test-curl-11053-e2d371"
Nov 03 20:06:09.144534 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:09.144502    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6sd\" (UniqueName: \"kubernetes.io/projected/b394a6ef-7d75-42ec-80f2-4c691cc71420-kube-api-access-wk6sd\") pod \"test-curl-11053-e2d371\" (UID: \"b394a6ef-7d75-42ec-80f2-4c691cc71420\") " pod="opendatahub/test-curl-11053-e2d371"
Nov 03 20:06:09.314865 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:09.313242    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11053-e2d371"
Nov 03 20:06:09.679692 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:09.679661    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:09.680806 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:09.680780    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:06:09.681029 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:09.681001    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 20:06:09.742585 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:09.742126    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:09.742585 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:09.742180    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:06:09.829104 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:09.829056    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11053-e2d371"]
Nov 03 20:06:09.847764 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:06:09.847723    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb394a6ef_7d75_42ec_80f2_4c691cc71420.slice/crio-86c9d6abc4bf7b9db59de010aaca2695516f8b55bc3160867dd3eb8ca8e7bd01 WatchSource:0}: Error finding container 86c9d6abc4bf7b9db59de010aaca2695516f8b55bc3160867dd3eb8ca8e7bd01: Status 404 returned error can't find the container with id 86c9d6abc4bf7b9db59de010aaca2695516f8b55bc3160867dd3eb8ca8e7bd01
Nov 03 20:06:10.654711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:10.654681    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:06:10.657333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:10.657309    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:06:10.679998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:10.679960    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:06:10.680187 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:10.680041    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 20:06:10.774306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:10.774267    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11053-e2d371" event={"ID":"b394a6ef-7d75-42ec-80f2-4c691cc71420","Type":"ContainerStarted","Data":"86c9d6abc4bf7b9db59de010aaca2695516f8b55bc3160867dd3eb8ca8e7bd01"}
Nov 03 20:06:10.859349 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:10.859320    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:06:10.860856 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:10.860833    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:06:11.680222 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:11.680190    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:06:11.680611 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:11.680242    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 20:06:11.795462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:11.783517    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11053-e2d371" event={"ID":"b394a6ef-7d75-42ec-80f2-4c691cc71420","Type":"ContainerStarted","Data":"520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad"}
Nov 03 20:06:11.812458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:11.812396    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11053-e2d371" podStartSLOduration=2.729210027 podStartE2EDuration="3.812382153s" totalImagesPullingTime="1.083172126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:11.807924913 +0000 UTC m=+1554.320746696" watchObservedRunningTime="2026-11-03 20:06:11.812382153 +0000 UTC m=+1554.325203935"
Nov 03 20:06:12.539398 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.539285    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11053-e2d371"]
Nov 03 20:06:12.561300 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.546370    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11053-e2d371"]
Nov 03 20:06:12.680017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.679984    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:06:12.680193 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.680054    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 20:06:12.716288 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.716252    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11053-932f7f"]
Nov 03 20:06:12.716923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.716906    5600 cpu_manager.go:414] "containerMap: removing container" podUID="b394a6ef-7d75-42ec-80f2-4c691cc71420" containerName="test-curl-11053-e2d371"
Nov 03 20:06:12.716971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.716924    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b394a6ef-7d75-42ec-80f2-4c691cc71420" containerName="test-curl-11053-e2d371"
Nov 03 20:06:12.717054 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.717045    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="b394a6ef-7d75-42ec-80f2-4c691cc71420" containerName="test-curl-11053-e2d371"
Nov 03 20:06:12.720765 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.720745    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11053-932f7f"
Nov 03 20:06:12.731907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.731614    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11053-932f7f"]
Nov 03 20:06:12.823490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.823380    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kthvz\" (UniqueName: \"kubernetes.io/projected/0f64b2e9-049c-4fcb-8549-69bef6120734-kube-api-access-kthvz\") pod \"test-curl-11053-932f7f\" (UID: \"0f64b2e9-049c-4fcb-8549-69bef6120734\") " pod="opendatahub/test-curl-11053-932f7f"
Nov 03 20:06:12.925117 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.925067    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kthvz\" (UniqueName: \"kubernetes.io/projected/0f64b2e9-049c-4fcb-8549-69bef6120734-kube-api-access-kthvz\") pod \"test-curl-11053-932f7f\" (UID: \"0f64b2e9-049c-4fcb-8549-69bef6120734\") " pod="opendatahub/test-curl-11053-932f7f"
Nov 03 20:06:12.934381 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:12.934357    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kthvz\" (UniqueName: \"kubernetes.io/projected/0f64b2e9-049c-4fcb-8549-69bef6120734-kube-api-access-kthvz\") pod \"test-curl-11053-932f7f\" (UID: \"0f64b2e9-049c-4fcb-8549-69bef6120734\") " pod="opendatahub/test-curl-11053-932f7f"
Nov 03 20:06:13.038114 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:13.038083    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11053-932f7f"
Nov 03 20:06:13.356575 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:06:13.356540    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f64b2e9_049c_4fcb_8549_69bef6120734.slice/crio-a0402ede5515e7dd8523f100d1a5d6dc0d4903127cda1873d1ebd999ab80e61a WatchSource:0}: Error finding container a0402ede5515e7dd8523f100d1a5d6dc0d4903127cda1873d1ebd999ab80e61a: Status 404 returned error can't find the container with id a0402ede5515e7dd8523f100d1a5d6dc0d4903127cda1873d1ebd999ab80e61a
Nov 03 20:06:13.356958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:13.356932    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11053-932f7f"]
Nov 03 20:06:13.680381 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:13.680348    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:06:13.680599 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:13.680397    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 20:06:13.833985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:13.832778    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11053-e2d371" podUID="b394a6ef-7d75-42ec-80f2-4c691cc71420" containerName="test-curl-11053-e2d371" containerID="cri-o://520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad" gracePeriod=2
Nov 03 20:06:13.833985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:13.833731    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11053-932f7f" event={"ID":"0f64b2e9-049c-4fcb-8549-69bef6120734","Type":"ContainerStarted","Data":"2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383"}
Nov 03 20:06:13.833985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:13.833765    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11053-932f7f" event={"ID":"0f64b2e9-049c-4fcb-8549-69bef6120734","Type":"ContainerStarted","Data":"a0402ede5515e7dd8523f100d1a5d6dc0d4903127cda1873d1ebd999ab80e61a"}
Nov 03 20:06:13.861548 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:13.861496    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11053-932f7f" podStartSLOduration=1.8614797429999999 podStartE2EDuration="1.861479743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:13.859146913 +0000 UTC m=+1556.371968696" watchObservedRunningTime="2026-11-03 20:06:13.861479743 +0000 UTC m=+1556.374301525"
Nov 03 20:06:14.679966 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:14.679923    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:06:14.680167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:14.679992    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 20:06:14.818561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:14.796492    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11053-932f7f"]
Nov 03 20:06:14.818561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:14.817954    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11053-932f7f"]
Nov 03 20:06:15.679921 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:15.679888    5600 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 20:06:15.680348 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:15.679949    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 20:06:15.867381 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:15.867341    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11053-932f7f" podUID="0f64b2e9-049c-4fcb-8549-69bef6120734" containerName="test-curl-11053-932f7f" containerID="cri-o://2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383" gracePeriod=2
Nov 03 20:06:16.220851 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.220813    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"]
Nov 03 20:06:16.225261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.225234    5600 cpu_manager.go:414] "containerMap: removing container" podUID="0f64b2e9-049c-4fcb-8549-69bef6120734" containerName="test-curl-11053-932f7f"
Nov 03 20:06:16.230889 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.230860    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f64b2e9-049c-4fcb-8549-69bef6120734" containerName="test-curl-11053-932f7f"
Nov 03 20:06:16.231587 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.231564    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f64b2e9-049c-4fcb-8549-69bef6120734" containerName="test-curl-11053-932f7f"
Nov 03 20:06:16.244023 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.243994    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.262792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.262711    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8fdab3c3-openshift-default-dockercfg-lp6wl\""
Nov 03 20:06:16.265611 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.264649    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11053-e2d371"
Nov 03 20:06:16.276065 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.276028    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"]
Nov 03 20:06:16.384588 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.365992    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b394a6ef-7d75-42ec-80f2-4c691cc71420-kube-api-access-wk6sd\" (UniqueName: \"kubernetes.io/projected/b394a6ef-7d75-42ec-80f2-4c691cc71420-kube-api-access-wk6sd\") pod \"b394a6ef-7d75-42ec-80f2-4c691cc71420\" (UID: \"b394a6ef-7d75-42ec-80f2-4c691cc71420\") "
Nov 03 20:06:16.394380 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.394326    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-data\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.398941 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.398909    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-token\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.399390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.399360    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tss4s\" (UniqueName: \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-kube-api-access-tss4s\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.399654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.399634    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istiod-ca-cert\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.399993 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.399973    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-podinfo\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.400224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.400205    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-certs\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.400631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.400612    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-envoy\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.400877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.400858    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-credential-socket\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.411786 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.411748    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-socket\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.445676 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.445642    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b394a6ef-7d75-42ec-80f2-4c691cc71420-kube-api-access-wk6sd" pod "b394a6ef-7d75-42ec-80f2-4c691cc71420" (UID: "b394a6ef-7d75-42ec-80f2-4c691cc71420"). InnerVolumeSpecName "kube-api-access-wk6sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:16.516180 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.516057    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istiod-ca-cert\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.517475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.517452    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-podinfo\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.517692 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.517675    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-certs\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.517977 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.517960    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-envoy\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.532569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.531906    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-certs\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.533190 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.517320    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istiod-ca-cert\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.533345 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.532780    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-credential-socket\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.533603 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.533120    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-credential-socket\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.533808 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.533771    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-socket\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.534365 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.534239    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-socket\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.534586 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.534547    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-data\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.535077 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.535050    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-token\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.535937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.535630    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tss4s\" (UniqueName: \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-kube-api-access-tss4s\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.536214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.534983    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-data\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.536405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.536390    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk6sd\" (UniqueName: \"kubernetes.io/projected/b394a6ef-7d75-42ec-80f2-4c691cc71420-kube-api-access-wk6sd\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:16.552795 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.552726    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-envoy\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.553465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.553417    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-podinfo\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.564499 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.564391    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tss4s\" (UniqueName: \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-kube-api-access-tss4s\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.565158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.565099    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-token\") pod \"e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") " pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.580160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.579737    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:16.787405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.785423    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:16.787405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.787224    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:16.791027 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.790998    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:06:16.876522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.876484    5600 generic.go:356] "Generic (PLEG): container finished" podID="b394a6ef-7d75-42ec-80f2-4c691cc71420" containerID="520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad" exitCode=137
Nov 03 20:06:16.876694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.876569    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11053-e2d371"
Nov 03 20:06:16.876694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.876625    5600 scope.go:122] "RemoveContainer" containerID="520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad"
Nov 03 20:06:16.889870 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.889850    5600 scope.go:122] "RemoveContainer" containerID="520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad"
Nov 03 20:06:16.890219 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:06:16.890199    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad\": container with ID starting with 520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad not found: ID does not exist" containerID="520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad"
Nov 03 20:06:16.890285 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:16.890237    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad"} err="failed to get container status \"520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad\": rpc error: code = NotFound desc = could not find container \"520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad\": container with ID starting with 520f9c59547bfc6abb9f9ee8a571714a1f059db917ad96a0602f0ec585986cad not found: ID does not exist"
Nov 03 20:06:16.928573 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:06:16.928523    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11053-e2d371\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="b394a6ef-7d75-42ec-80f2-4c691cc71420" pod="opendatahub/test-curl-11053-e2d371"
Nov 03 20:06:17.090299 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:17.090257    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"]
Nov 03 20:06:17.101633 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:17.101594    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:06:17.101799 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:17.101689    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:06:17.101799 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:17.101723    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:06:17.616626 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:06:17.616581    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11053-e2d371\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="b394a6ef-7d75-42ec-80f2-4c691cc71420" pod="opendatahub/test-curl-11053-e2d371"
Nov 03 20:06:17.622035 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:17.622001    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b394a6ef-7d75-42ec-80f2-4c691cc71420" path="/var/lib/kubelet/pods/b394a6ef-7d75-42ec-80f2-4c691cc71420/volumes"
Nov 03 20:06:17.887617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:17.887584    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq" event={"ID":"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd","Type":"ContainerStarted","Data":"1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef"}
Nov 03 20:06:17.888048 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:17.887665    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq" event={"ID":"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd","Type":"ContainerStarted","Data":"6144fe90e0275c0f9e013641491f767ecae76e96d3d6d8231b9182c7a15c151a"}
Nov 03 20:06:17.937909 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:17.937858    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq" podStartSLOduration=1.937837457 podStartE2EDuration="1.937837457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:17.932538805 +0000 UTC m=+1560.445360588" watchObservedRunningTime="2026-11-03 20:06:17.937837457 +0000 UTC m=+1560.450659239"
Nov 03 20:06:18.194843 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.194808    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11053-932f7f"
Nov 03 20:06:18.306196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.306160    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f64b2e9-049c-4fcb-8549-69bef6120734-kube-api-access-kthvz\" (UniqueName: \"kubernetes.io/projected/0f64b2e9-049c-4fcb-8549-69bef6120734-kube-api-access-kthvz\") pod \"0f64b2e9-049c-4fcb-8549-69bef6120734\" (UID: \"0f64b2e9-049c-4fcb-8549-69bef6120734\") "
Nov 03 20:06:18.308939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.308907    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f64b2e9-049c-4fcb-8549-69bef6120734-kube-api-access-kthvz" pod "0f64b2e9-049c-4fcb-8549-69bef6120734" (UID: "0f64b2e9-049c-4fcb-8549-69bef6120734"). InnerVolumeSpecName "kube-api-access-kthvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:18.408981 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.408899    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kthvz\" (UniqueName: \"kubernetes.io/projected/0f64b2e9-049c-4fcb-8549-69bef6120734-kube-api-access-kthvz\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:18.581468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.580842    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:18.713516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.706650    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:18.900651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.900612    5600 generic.go:356] "Generic (PLEG): container finished" podID="0f64b2e9-049c-4fcb-8549-69bef6120734" containerID="2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383" exitCode=137
Nov 03 20:06:18.901160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.900662    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11053-932f7f"
Nov 03 20:06:18.901160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.900741    5600 scope.go:122] "RemoveContainer" containerID="2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383"
Nov 03 20:06:18.901625 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.901476    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:18.904015 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.903993    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:06:18.914861 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.914825    5600 scope.go:122] "RemoveContainer" containerID="2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383"
Nov 03 20:06:18.915222 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:06:18.915185    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383\": container with ID starting with 2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383 not found: ID does not exist" containerID="2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383"
Nov 03 20:06:18.915316 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:18.915234    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383"} err="failed to get container status \"2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383\": rpc error: code = NotFound desc = could not find container \"2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383\": container with ID starting with 2e8e9cbdf5375225f3f8d36e153b2441f44a3e38cc3a72a47a39df3aad0b5383 not found: ID does not exist"
Nov 03 20:06:18.945347 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:06:18.945217    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11053-932f7f\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="0f64b2e9-049c-4fcb-8549-69bef6120734" pod="opendatahub/test-curl-11053-932f7f"
Nov 03 20:06:19.671257 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:19.663954    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f64b2e9-049c-4fcb-8549-69bef6120734" path="/var/lib/kubelet/pods/0f64b2e9-049c-4fcb-8549-69bef6120734/volumes"
Nov 03 20:06:28.798605 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:06:28.798560    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:06:29.222239 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.222207    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"]
Nov 03 20:06:29.231163 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.231137    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.238394 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.238366    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"]
Nov 03 20:06:29.238639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.238375    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8fdab3c3-serving-cert\""
Nov 03 20:06:29.356530 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.356486    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-metrics-certs\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.356765 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.356633    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdr7c\" (UniqueName: \"kubernetes.io/projected/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-kube-api-access-mdr7c\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.356855 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.356788    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-openshift-service-ca\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.356922 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.356887    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-maas-api-tls\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.458335 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.458289    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdr7c\" (UniqueName: \"kubernetes.io/projected/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-kube-api-access-mdr7c\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.458596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.458374    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-openshift-service-ca\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.458596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.458467    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-maas-api-tls\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.458596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.458554    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-metrics-certs\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.459196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.459174    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-openshift-service-ca\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.462004 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.461978    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-metrics-certs\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.462567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.462545    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-maas-api-tls\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.467792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.467767    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdr7c\" (UniqueName: \"kubernetes.io/projected/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-kube-api-access-mdr7c\") pod \"maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.555535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.555419    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:29.924150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:29.924106    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"]
Nov 03 20:06:30.107788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:30.107748    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb" event={"ID":"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d","Type":"ContainerStarted","Data":"bd70eb74e2cf2045df6759d8286aa0f45913e49b7b3c55ba2078fc9211163af8"}
Nov 03 20:06:31.117043 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.117010    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb" event={"ID":"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d","Type":"ContainerStarted","Data":"47f1b11c690e8bc535848d8e4655e574cd0188f87b52cc13b2425f32c7460a9b"}
Nov 03 20:06:31.117510 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.117244    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:31.166042 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.165987    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb" podStartSLOduration=1.570022637 podStartE2EDuration="2.165973965s" totalImagesPullingTime="595.951328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:31.16269694 +0000 UTC m=+1573.675518723" watchObservedRunningTime="2026-11-03 20:06:31.165973965 +0000 UTC m=+1573.678795746"
Nov 03 20:06:31.742616 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.742580    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"]
Nov 03 20:06:31.752184 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.752155    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.757918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.756940    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"]
Nov 03 20:06:31.758304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.758276    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ea700cd3-serving-cert\""
Nov 03 20:06:31.809325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.809286    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblxc\" (UniqueName: \"kubernetes.io/projected/b20f7896-51a8-4d5a-8c9d-1466659572cc-kube-api-access-cblxc\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.809700 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.809676    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-metrics-certs\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.809982 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.809958    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20f7896-51a8-4d5a-8c9d-1466659572cc-openshift-service-ca\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.810099 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.810066    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-maas-api-tls\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.910979 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.910942    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblxc\" (UniqueName: \"kubernetes.io/projected/b20f7896-51a8-4d5a-8c9d-1466659572cc-kube-api-access-cblxc\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.911176 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.911030    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-metrics-certs\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.911283 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.911184    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20f7896-51a8-4d5a-8c9d-1466659572cc-openshift-service-ca\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.911342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.911317    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-maas-api-tls\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.911964 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.911939    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20f7896-51a8-4d5a-8c9d-1466659572cc-openshift-service-ca\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.913765 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.913744    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-metrics-certs\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.913856 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.913804    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-maas-api-tls\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:31.919119 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:31.919101    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblxc\" (UniqueName: \"kubernetes.io/projected/b20f7896-51a8-4d5a-8c9d-1466659572cc-kube-api-access-cblxc\") pod \"maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:32.071188 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:32.071106    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:32.366414 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:32.366381    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"]
Nov 03 20:06:32.367725 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:06:32.367692    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20f7896_51a8_4d5a_8c9d_1466659572cc.slice/crio-d47d7e0b96dd2f8233e84905309ea54aa878f3c28b29203ce03320399fb705e0 WatchSource:0}: Error finding container d47d7e0b96dd2f8233e84905309ea54aa878f3c28b29203ce03320399fb705e0: Status 404 returned error can't find the container with id d47d7e0b96dd2f8233e84905309ea54aa878f3c28b29203ce03320399fb705e0
Nov 03 20:06:33.186457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:33.182764    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89" event={"ID":"b20f7896-51a8-4d5a-8c9d-1466659572cc","Type":"ContainerStarted","Data":"d47d7e0b96dd2f8233e84905309ea54aa878f3c28b29203ce03320399fb705e0"}
Nov 03 20:06:33.940948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:33.937793    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:33.999455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:33.996826    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.009825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:34.008384    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:06:34.128539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:34.095877    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.306518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:34.305550    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:06:34.508099 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:34.507762    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:06:34.559207 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:34.559108    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89" event={"ID":"b20f7896-51a8-4d5a-8c9d-1466659572cc","Type":"ContainerStarted","Data":"ee3654f8321550b3c84dd3c6fbd8b592a6460a8e879f2bbb8ff4045ac7322123"}
Nov 03 20:06:34.577461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:34.561233    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:34.749189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:34.719914    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89" podStartSLOduration=3.133113188 podStartE2EDuration="3.719887414s" totalImagesPullingTime="586.774226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:34.604412061 +0000 UTC m=+1577.117233843" watchObservedRunningTime="2026-11-03 20:06:34.719887414 +0000 UTC m=+1577.232709198"
Nov 03 20:06:34.941589 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:34.941539    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.988463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:34.953223    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.988463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:34.981966    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:06:35.015463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:35.013136    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:35.015463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:35.013687    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:06:35.016725 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:35.016693    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:35.047581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:35.017473    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:06:35.118782 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:35.108206    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:35.123458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:35.119800    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:06:35.318909 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:35.316728    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:35.318909 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:35.317512    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:06:35.746492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:35.719650    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:06:36.040830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:36.016569    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.123680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:36.055560    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.148574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:36.119568    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:06:36.184870 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:36.184837    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:06:36.185062 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:36.184918    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:06:36.348677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:36.348579    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:06:36.720600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:36.720550    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.899608 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:36.899548    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:37.171658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:37.135139    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:37.171658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:37.163548    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.171658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:37.167076    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.184637 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:37.184538    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:37.307692 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:37.307658    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:06:37.414414 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:37.414369    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.742620 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:37.742575    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.901066 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:37.900520    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.020451 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.020342    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:06:38.086584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.086543    5600 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:38.086900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.086853    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.186462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.147548    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:38.186462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.179707    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.186462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.182007    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:06:38.235785 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.235737    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.324151 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.320849    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:06:38.423470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.411741    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 03 20:06:38.443466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.427033    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.757537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.757343    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.765464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.758764    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.910457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:38.909559    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:39.151574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:39.150532    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:39.188536 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:39.187544    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:39.778462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:39.775899    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:06:39.778685 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:39.778595    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:39.804494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:39.803675    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:39.942729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:39.918500    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.137612 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:40.137568    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:06:40.158613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:40.158559    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.205631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:40.205571    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:06:40.793589 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:40.793537    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.811464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:40.804062    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.968545 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:40.964955    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.139822 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:41.139775    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.141535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:41.141512    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:06:41.163667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:41.163626    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.246827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:41.246779    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.776753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:41.776687    5600 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:41.777110 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:41.777026    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.813315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:41.797996    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.839197 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:41.839133    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.954628 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:41.954582    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:41.954997 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:41.954950    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:42.048185 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:42.047601    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:06:42.246043 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:42.245991    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:42.380390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:42.274527    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:42.381455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:42.275891    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:42.381811 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:42.379257    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:06:42.850225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:42.848639    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:42.881478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:42.879029    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:43.026784 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:43.020505    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:06:43.113806 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:43.113694    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:06:43.153768 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:43.115774    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:06:43.296492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:43.296464    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:06:43.299876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:43.296767    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-dq9xl" podUID="adbf5e48-bc06-4df3-aa7d-fe2731c194f8" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:43.451011 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:43.450810    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:43.581126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:43.581089    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:06:43.582581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:43.582556    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:06:43.584632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:43.584605    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:06:55.571985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.571950    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"]
Nov 03 20:06:55.577917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.577896    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.583491 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.583470    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 20:06:55.586472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.586451    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 20:06:55.586574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.586491    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 20:06:55.586652 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.586640    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gjlpz\""
Nov 03 20:06:55.596495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.596471    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"]
Nov 03 20:06:55.677401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.677360    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.677584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.677510    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.677699 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.677669    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wf4\" (UniqueName: \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-kube-api-access-47wf4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.677837 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.677798    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.677899 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.677847    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.677948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.677926    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.678054 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.678038    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.779631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.779599    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wf4\" (UniqueName: \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-kube-api-access-47wf4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.779792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.779665    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.779792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.779702    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.779792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.779748    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.779958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.779834    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.779958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.779910    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.780064 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.779978    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.780547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.780523    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.782385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.782355    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.782552 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.782404    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.782601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.782565    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.783009 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.782988    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.788005 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.787982    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.788907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.788884    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wf4\" (UniqueName: \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-kube-api-access-47wf4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrs7z\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:55.892232 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:55.892197    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:56.188818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:56.188789    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"]
Nov 03 20:06:56.189726 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:06:56.189700    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e010c68_f9f3_48c9_b578_d61dbbbf92eb.slice/crio-c1448b4ba6800a3d8322b25bb5d64c98cdb5caa15cef5b1bbac79e708dc6e143 WatchSource:0}: Error finding container c1448b4ba6800a3d8322b25bb5d64c98cdb5caa15cef5b1bbac79e708dc6e143: Status 404 returned error can't find the container with id c1448b4ba6800a3d8322b25bb5d64c98cdb5caa15cef5b1bbac79e708dc6e143
Nov 03 20:06:56.853147 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:56.853111    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z" event={"ID":"5e010c68-f9f3-48c9-b578-d61dbbbf92eb","Type":"ContainerStarted","Data":"c1448b4ba6800a3d8322b25bb5d64c98cdb5caa15cef5b1bbac79e708dc6e143"}
Nov 03 20:06:58.809683 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:06:58.809623    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:06:59.346121 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:59.346081    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:06:59.346260 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:59.346158    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:06:59.881925 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:59.881887    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z" event={"ID":"5e010c68-f9f3-48c9-b578-d61dbbbf92eb","Type":"ContainerStarted","Data":"213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08"}
Nov 03 20:06:59.882427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:59.881967    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:06:59.909026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:06:59.908970    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z" podStartSLOduration=1.755723527 podStartE2EDuration="4.908950591s" totalImagesPullingTime="3.153227064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:59.90311593 +0000 UTC m=+1602.415937713" watchObservedRunningTime="2026-11-03 20:06:59.908950591 +0000 UTC m=+1602.421772374"
Nov 03 20:07:00.905456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:00.900236    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:07:07.286802 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:07.285744    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"]
Nov 03 20:07:07.286802 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:07.286034    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb" podUID="9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d" containerName="maas-api" containerID="cri-o://47f1b11c690e8bc535848d8e4655e574cd0188f87b52cc13b2425f32c7460a9b" gracePeriod=30
Nov 03 20:07:07.998470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:07.996890    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"]
Nov 03 20:07:08.004924 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:08.004845    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq" podUID="8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" containerName="istio-proxy" containerID="cri-o://1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef" gracePeriod=30
Nov 03 20:07:08.294507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:08.294386    5600 generic.go:356] "Generic (PLEG): container finished" podID="9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d" containerID="47f1b11c690e8bc535848d8e4655e574cd0188f87b52cc13b2425f32c7460a9b" exitCode=0
Nov 03 20:07:08.295288 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:08.295255    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb" event={"ID":"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d","Type":"ContainerDied","Data":"47f1b11c690e8bc535848d8e4655e574cd0188f87b52cc13b2425f32c7460a9b"}
Nov 03 20:07:08.408081 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:08.408033    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 03 20:07:08.862937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:08.862890    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:08.930605 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:08.930560    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:08.989459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:08.983794    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:07:09.041769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.040961    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.155669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.155547    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.199617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.199524    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.300468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.300211    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:07:09.374463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.373093    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb" event={"ID":"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d","Type":"ContainerDied","Data":"bd70eb74e2cf2045df6759d8286aa0f45913e49b7b3c55ba2078fc9211163af8"}
Nov 03 20:07:09.374463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.373170    5600 scope.go:122] "RemoveContainer" containerID="47f1b11c690e8bc535848d8e4655e574cd0188f87b52cc13b2425f32c7460a9b"
Nov 03 20:07:09.374463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.373330    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"
Nov 03 20:07:09.425503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.422089    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-openshift-service-ca\") pod \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") "
Nov 03 20:07:09.425503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.422297    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-metrics-certs\") pod \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") "
Nov 03 20:07:09.425503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.422376    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-kube-api-access-mdr7c\" (UniqueName: \"kubernetes.io/projected/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-kube-api-access-mdr7c\") pod \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") "
Nov 03 20:07:09.425503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.422461    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-maas-api-tls\") pod \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\" (UID: \"9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d\") "
Nov 03 20:07:09.532462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.526609    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-metrics-certs" pod "9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d" (UID: "9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:09.590448 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.574353    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:09.657001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.597680    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-openshift-service-ca" pod "9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d" (UID: "9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:09.657001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.598360    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-maas-api-tls" pod "9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d" (UID: "9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:09.785502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.703617    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:09.810543 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.810504    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:09.975617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.975570    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.535332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.013329    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:10.535332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:09.744819    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.535332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.063102    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:07:10.535332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.082596    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-kube-api-access-mdr7c" pod "9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d" (UID: "9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d"). InnerVolumeSpecName "kube-api-access-mdr7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:10.535332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.082694    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.535332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.156357    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.535332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.205238    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.565415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.565217    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdr7c\" (UniqueName: \"kubernetes.io/projected/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d-kube-api-access-mdr7c\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:10.802515 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:07:10.802100    5600 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.183s"
Nov 03 20:07:10.802515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.802301    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:07:10.802515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.802356    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:07:10.802515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.802408    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:07:10.802515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.802478    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:07:10.802924 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.802528    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:07:10.802924 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.802573    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:07:10.957476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:10.956176    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"]
Nov 03 20:07:11.217461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:11.217012    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fdab3c3-7fc74fddd5-djgqb"]
Nov 03 20:07:11.544991 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:11.544871    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:11.545976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:11.545939    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:07:11.618748 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:11.555088    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:11.619409 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:11.555143    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:07:11.620254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:11.555165    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:07:11.737535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:11.718248    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:12.037684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:12.037643    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:07:12.038038 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:12.037985    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:12.129461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:12.089830    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d" path="/var/lib/kubelet/pods/9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d/volumes"
Nov 03 20:07:12.551665 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:12.551616    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:12.632684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:12.632639    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:12.633321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:12.633283    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:12.633848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:12.633827    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:07:12.634255 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:12.634235    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:07:12.728388 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:12.728342    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.556464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:13.556372    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.645464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:13.639130    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.645464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:13.639554    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.645464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:13.639818    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.645464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:13.640040    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.738344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:13.738303    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:14.147461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:14.146791    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:07:14.569261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:14.558410    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:14.650474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:14.644572    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:14.650474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:14.644972    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:07:14.650474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:14.645269    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:14.796458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:14.760576    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:14.940463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:14.938907    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:15.565800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:15.565751    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:15.715821 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:15.715706    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:15.718200 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:15.718152    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:15.797457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:15.777759    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:15.797457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:15.730649    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:07:15.797457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:15.719227    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:07:16.572595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:16.572540    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:16.728944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:16.715844    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"]
Nov 03 20:07:16.728944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:16.716100    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89" podUID="b20f7896-51a8-4d5a-8c9d-1466659572cc" containerName="maas-api" containerID="cri-o://ee3654f8321550b3c84dd3c6fbd8b592a6460a8e879f2bbb8ff4045ac7322123" gracePeriod=30
Nov 03 20:07:16.728944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:16.716506    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:16.822466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:16.800976    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:16.822466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:16.803054    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:16.822466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:16.803278    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:16.822466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:16.803855    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:07:16.823907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:16.823484    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:17.206959 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:17.195600    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:17.315053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:17.314981    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89" podUID="b20f7896-51a8-4d5a-8c9d-1466659572cc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.84:8443/health\": dial tcp 10.134.0.84:8443: connect: connection refused"
Nov 03 20:07:17.587529 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:17.585965    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:17.722661 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:17.718536    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:17.847684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:17.812606    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:17.847684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:17.813047    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:07:17.847684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:17.813514    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:07:17.861501 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:17.848953    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:18.235548 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.222992    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:18.401013 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.400973    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 03 20:07:18.617020 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.586367    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:18.829562 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.827528    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:18.829562 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.827854    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:18.865481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.848128    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:07:18.865481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.849036    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:07:18.865481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.849838    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:18.917412 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.889634    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:18.925647 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.925615    5600 generic.go:356] "Generic (PLEG): container finished" podID="b20f7896-51a8-4d5a-8c9d-1466659572cc" containerID="ee3654f8321550b3c84dd3c6fbd8b592a6460a8e879f2bbb8ff4045ac7322123" exitCode=0
Nov 03 20:07:18.925840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.925822    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89" event={"ID":"b20f7896-51a8-4d5a-8c9d-1466659572cc","Type":"ContainerDied","Data":"ee3654f8321550b3c84dd3c6fbd8b592a6460a8e879f2bbb8ff4045ac7322123"}
Nov 03 20:07:18.959815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.950955    5600 patch_prober.go:29] interesting pod/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:07:18.959815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:18.951013    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq" podUID="8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:07:19.215072 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.215039    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:07:19.344590 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.344468    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-metrics-certs\") pod \"b20f7896-51a8-4d5a-8c9d-1466659572cc\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") "
Nov 03 20:07:19.344879 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.344859    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-maas-api-tls\") pod \"b20f7896-51a8-4d5a-8c9d-1466659572cc\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") "
Nov 03 20:07:19.345068 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.345045    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20f7896-51a8-4d5a-8c9d-1466659572cc-kube-api-access-cblxc\" (UniqueName: \"kubernetes.io/projected/b20f7896-51a8-4d5a-8c9d-1466659572cc-kube-api-access-cblxc\") pod \"b20f7896-51a8-4d5a-8c9d-1466659572cc\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") "
Nov 03 20:07:19.345551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.345517    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20f7896-51a8-4d5a-8c9d-1466659572cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20f7896-51a8-4d5a-8c9d-1466659572cc-openshift-service-ca\") pod \"b20f7896-51a8-4d5a-8c9d-1466659572cc\" (UID: \"b20f7896-51a8-4d5a-8c9d-1466659572cc\") "
Nov 03 20:07:19.381841 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.381697    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20f7896-51a8-4d5a-8c9d-1466659572cc-openshift-service-ca" pod "b20f7896-51a8-4d5a-8c9d-1466659572cc" (UID: "b20f7896-51a8-4d5a-8c9d-1466659572cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:19.555460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.526271    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20f7896-51a8-4d5a-8c9d-1466659572cc-kube-api-access-cblxc" pod "b20f7896-51a8-4d5a-8c9d-1466659572cc" (UID: "b20f7896-51a8-4d5a-8c9d-1466659572cc"). InnerVolumeSpecName "kube-api-access-cblxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:19.594784 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.587727    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:19.627475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.621486    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cblxc\" (UniqueName: \"kubernetes.io/projected/b20f7896-51a8-4d5a-8c9d-1466659572cc-kube-api-access-cblxc\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:19.627475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.621520    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b20f7896-51a8-4d5a-8c9d-1466659572cc-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:19.757627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.757564    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-maas-api-tls" pod "b20f7896-51a8-4d5a-8c9d-1466659572cc" (UID: "b20f7896-51a8-4d5a-8c9d-1466659572cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:19.842461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.827813    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:19.842461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.828142    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:19.848819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.848727    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:19.893481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.884955    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:07:19.893481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.884999    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:07:19.893481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.885021    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:07:19.996252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:19.995543    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:20.151481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.150565    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-metrics-certs" pod "b20f7896-51a8-4d5a-8c9d-1466659572cc" (UID: "b20f7896-51a8-4d5a-8c9d-1466659572cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:20.173459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.171945    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b20f7896-51a8-4d5a-8c9d-1466659572cc-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:20.249459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.248934    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:07:20.394414 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.394383    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:07:20.451462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.451348    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89" event={"ID":"b20f7896-51a8-4d5a-8c9d-1466659572cc","Type":"ContainerDied","Data":"d47d7e0b96dd2f8233e84905309ea54aa878f3c28b29203ce03320399fb705e0"}
Nov 03 20:07:20.458579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.458550    5600 scope.go:122] "RemoveContainer" containerID="ee3654f8321550b3c84dd3c6fbd8b592a6460a8e879f2bbb8ff4045ac7322123"
Nov 03 20:07:20.459211 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.459189    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"
Nov 03 20:07:20.841014 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.840986    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"]
Nov 03 20:07:20.841331 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.841303    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:20.846262 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.846235    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea700cd3-5868fbf688-2mh89"]
Nov 03 20:07:20.892120 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.892077    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:20.893980 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.892351    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:07:20.934116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:20.934062    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:21.003196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:21.001573    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:07:21.270610 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:21.266618    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:07:21.270610 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:21.267768    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:07:21.272346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:21.272310    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:07:21.690779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:21.682152    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20f7896-51a8-4d5a-8c9d-1466659572cc" path="/var/lib/kubelet/pods/b20f7896-51a8-4d5a-8c9d-1466659572cc/volumes"
Nov 03 20:07:28.841586 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:07:28.819660    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:07:29.192308 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.192276    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:07:29.254795 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.254764    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-data\") pod \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") "
Nov 03 20:07:29.254981 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.254929    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-kube-api-access-tss4s\" (UniqueName: \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-kube-api-access-tss4s\") pod \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") "
Nov 03 20:07:29.255050 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.255000    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-envoy\") pod \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") "
Nov 03 20:07:29.255111 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.255085    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-socket\") pod \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") "
Nov 03 20:07:29.255213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.255198    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istiod-ca-cert\") pod \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") "
Nov 03 20:07:29.255286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.255271    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-token\" (UniqueName: \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-token\") pod \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") "
Nov 03 20:07:29.255342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.255316    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-credential-socket\") pod \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") "
Nov 03 20:07:29.255396 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.255372    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-certs\") pod \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") "
Nov 03 20:07:29.255531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.255518    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-podinfo\") pod \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\" (UID: \"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd\") "
Nov 03 20:07:29.256608 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.256573    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-socket" pod "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" (UID: "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:29.257507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.257422    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-credential-socket" pod "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" (UID: "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:29.257825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.257788    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istiod-ca-cert" pod "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" (UID: "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:29.257995 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.257964    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-certs" pod "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" (UID: "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:29.258463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.258419    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-data" pod "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" (UID: "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:29.259400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.259310    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-podinfo" pod "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" (UID: "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:07:29.261088 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.261063    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-token" pod "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" (UID: "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:29.261202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.261099    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-kube-api-access-tss4s" pod "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" (UID: "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd"). InnerVolumeSpecName "kube-api-access-tss4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:29.261859 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.261835    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-envoy" pod "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" (UID: "8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:29.357467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.357350    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:29.357467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.357384    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:29.357467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.357394    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tss4s\" (UniqueName: \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-kube-api-access-tss4s\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:29.357467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.357406    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:29.357467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.357416    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:29.357467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.357426    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:29.357785 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.357480    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:29.357785 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.357501    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:29.357785 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.357509    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:29.593276 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.593239    5600 generic.go:356] "Generic (PLEG): container finished" podID="8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" containerID="1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef" exitCode=0
Nov 03 20:07:29.593513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.593309    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"
Nov 03 20:07:29.593513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.593313    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq" event={"ID":"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd","Type":"ContainerDied","Data":"1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef"}
Nov 03 20:07:29.593513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.593353    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq" event={"ID":"8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd","Type":"ContainerDied","Data":"6144fe90e0275c0f9e013641491f767ecae76e96d3d6d8231b9182c7a15c151a"}
Nov 03 20:07:29.593513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.593379    5600 scope.go:122] "RemoveContainer" containerID="1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef"
Nov 03 20:07:29.605645 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.605623    5600 scope.go:122] "RemoveContainer" containerID="1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef"
Nov 03 20:07:29.605913 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:07:29.605893    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef\": container with ID starting with 1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef not found: ID does not exist" containerID="1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef"
Nov 03 20:07:29.605982 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.605922    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef"} err="failed to get container status \"1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef\": rpc error: code = NotFound desc = could not find container \"1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef\": container with ID starting with 1231718d1568611ad0cd857017dfe22c7e88b3d961318943637f61980f901aef not found: ID does not exist"
Nov 03 20:07:29.627123 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.627024    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"]
Nov 03 20:07:29.638543 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:29.637138    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8fdab3c3-openshift-default-74d6465cf7-hjkgq"]
Nov 03 20:07:30.382294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.382237    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"]
Nov 03 20:07:30.384304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.384279    5600 cpu_manager.go:414] "containerMap: removing container" podUID="8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" containerName="istio-proxy"
Nov 03 20:07:30.384511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.384496    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" containerName="istio-proxy"
Nov 03 20:07:30.384722 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.384711    5600 cpu_manager.go:414] "containerMap: removing container" podUID="9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d" containerName="maas-api"
Nov 03 20:07:30.384801 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.384791    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d" containerName="maas-api"
Nov 03 20:07:30.384901 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.384891    5600 cpu_manager.go:414] "containerMap: removing container" podUID="b20f7896-51a8-4d5a-8c9d-1466659572cc" containerName="maas-api"
Nov 03 20:07:30.384967 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.384958    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20f7896-51a8-4d5a-8c9d-1466659572cc" containerName="maas-api"
Nov 03 20:07:30.385868 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.385848    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd05e4a-ca2e-4fce-b89f-2c8e5b43062d" containerName="maas-api"
Nov 03 20:07:30.386026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.386013    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20f7896-51a8-4d5a-8c9d-1466659572cc" containerName="maas-api"
Nov 03 20:07:30.386146 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.386134    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" containerName="istio-proxy"
Nov 03 20:07:30.397100 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.397073    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.404510 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.403853    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"]
Nov 03 20:07:30.404510 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.404154    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-d81d18-serving-cert\""
Nov 03 20:07:30.481217 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.481183    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqdg\" (UniqueName: \"kubernetes.io/projected/87da290c-f102-4519-8977-018368b52685-kube-api-access-hqqdg\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.481410 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.481239    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.481410 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.481323    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.481549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.481416    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87da290c-f102-4519-8977-018368b52685-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.583633 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.583589    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.599505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.584417    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.599838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.599801    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87da290c-f102-4519-8977-018368b52685-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.602876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.601145    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqdg\" (UniqueName: \"kubernetes.io/projected/87da290c-f102-4519-8977-018368b52685-kube-api-access-hqqdg\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.602876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.600695    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87da290c-f102-4519-8977-018368b52685-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.602876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.597646    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.605557 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.604352    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.612051 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.612029    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqdg\" (UniqueName: \"kubernetes.io/projected/87da290c-f102-4519-8977-018368b52685-kube-api-access-hqqdg\") pod \"maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7\" (UID: \"87da290c-f102-4519-8977-018368b52685\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:30.734762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:30.722505    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:31.126616 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:31.120795    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"]
Nov 03 20:07:31.151484 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:07:31.148510    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87da290c_f102_4519_8977_018368b52685.slice/crio-9c1cd855c53cb52d917f4b687b4c63cef70aa52b54e271c2e34db68ea6689d52 WatchSource:0}: Error finding container 9c1cd855c53cb52d917f4b687b4c63cef70aa52b54e271c2e34db68ea6689d52: Status 404 returned error can't find the container with id 9c1cd855c53cb52d917f4b687b4c63cef70aa52b54e271c2e34db68ea6689d52
Nov 03 20:07:31.622457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:31.620471    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd" path="/var/lib/kubelet/pods/8d8b3610-6f8b-4076-b8c8-55ab41bc5bdd/volumes"
Nov 03 20:07:31.626604 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:31.626568    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7" event={"ID":"87da290c-f102-4519-8977-018368b52685","Type":"ContainerStarted","Data":"9c1cd855c53cb52d917f4b687b4c63cef70aa52b54e271c2e34db68ea6689d52"}
Nov 03 20:07:32.674091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:32.674050    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7" event={"ID":"87da290c-f102-4519-8977-018368b52685","Type":"ContainerStarted","Data":"0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36"}
Nov 03 20:07:32.675545 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:32.675520    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:38.688011 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:38.687980    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:07:38.710164 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:38.709200    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7" podStartSLOduration=8.227553913 podStartE2EDuration="8.709182285s" totalImagesPullingTime="481.628372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:32.753370605 +0000 UTC m=+1635.266192388" watchObservedRunningTime="2026-11-03 20:07:38.709182285 +0000 UTC m=+1641.222004068"
Nov 03 20:07:56.425623 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.425588    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"]
Nov 03 20:07:56.431242 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.431217    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.442001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.441974    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-944e83-serving-cert\""
Nov 03 20:07:56.447628 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.447601    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"]
Nov 03 20:07:56.593417 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.593376    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.593649 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.593619    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31ccaed-b4ee-416a-b88c-3ceefecd645a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.593727 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.593702    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tmzf\" (UniqueName: \"kubernetes.io/projected/e31ccaed-b4ee-416a-b88c-3ceefecd645a-kube-api-access-8tmzf\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.593946 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.593925    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.695673 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.695572    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.695858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.695743    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31ccaed-b4ee-416a-b88c-3ceefecd645a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.695858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.695807    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tmzf\" (UniqueName: \"kubernetes.io/projected/e31ccaed-b4ee-416a-b88c-3ceefecd645a-kube-api-access-8tmzf\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.695996 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.695976    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.696700 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.696673    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31ccaed-b4ee-416a-b88c-3ceefecd645a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.698960 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.698929    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.699393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.699369    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.704989 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.704942    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tmzf\" (UniqueName: \"kubernetes.io/projected/e31ccaed-b4ee-416a-b88c-3ceefecd645a-kube-api-access-8tmzf\") pod \"maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:56.760106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:56.760069    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:57.110406 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:57.110311    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"]
Nov 03 20:07:57.142419 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:07:57.142376    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31ccaed_b4ee_416a_b88c_3ceefecd645a.slice/crio-7130619ee05e04978d1d5fe0e485077082500d83fb1e2ee484ddb35be0cefadc WatchSource:0}: Error finding container 7130619ee05e04978d1d5fe0e485077082500d83fb1e2ee484ddb35be0cefadc: Status 404 returned error can't find the container with id 7130619ee05e04978d1d5fe0e485077082500d83fb1e2ee484ddb35be0cefadc
Nov 03 20:07:58.093232 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:58.093138    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv" event={"ID":"e31ccaed-b4ee-416a-b88c-3ceefecd645a","Type":"ContainerStarted","Data":"7130619ee05e04978d1d5fe0e485077082500d83fb1e2ee484ddb35be0cefadc"}
Nov 03 20:07:58.800652 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:07:58.800614    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:07:59.106165 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:59.106069    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv" event={"ID":"e31ccaed-b4ee-416a-b88c-3ceefecd645a","Type":"ContainerStarted","Data":"44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50"}
Nov 03 20:07:59.106165 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:59.106146    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:07:59.123981 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:07:59.123927    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv" podStartSLOduration=2.427557489 podStartE2EDuration="3.123909266s" totalImagesPullingTime="696.351777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:59.122642764 +0000 UTC m=+1661.635464549" watchObservedRunningTime="2026-11-03 20:07:59.123909266 +0000 UTC m=+1661.636731049"
Nov 03 20:08:01.229495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.228181    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"]
Nov 03 20:08:01.236903 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.236862    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.318818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.279781    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-d81d18-openshift-default-dockercfg-7vnl7\""
Nov 03 20:08:01.372788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.334460    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.372788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.334563    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.372788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.334722    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5789c64e-f731-4af7-96c0-f044fd64aaf6-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.372788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.334758    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.372788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.334823    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.372788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.334913    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.372788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.334943    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.372788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.335061    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.372788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.335096    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7zg6\" (UniqueName: \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-kube-api-access-f7zg6\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.372788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.352508    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"]
Nov 03 20:08:01.437984 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.437922    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5789c64e-f731-4af7-96c0-f044fd64aaf6-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.438939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.438913    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.439069 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.438990    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.439128 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.439079    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.439128 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.439111    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.439293 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.439227    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.439293 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.439264    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7zg6\" (UniqueName: \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-kube-api-access-f7zg6\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.439398 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.439303    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.439398 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.439381    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.439784 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.439733    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.440059 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.439990    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.440059 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.438849    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5789c64e-f731-4af7-96c0-f044fd64aaf6-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.442333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.441293    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.442333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.441613    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.444003 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.443954    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.446794 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.446751    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.466388 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.466320    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.473386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.473350    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7zg6\" (UniqueName: \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-kube-api-access-f7zg6\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:01.726898 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:01.726629    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:02.281238 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:02.281208    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"]
Nov 03 20:08:02.282748 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:08:02.282712    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5789c64e_f731_4af7_96c0_f044fd64aaf6.slice/crio-8ae1e077da559a0ce02c919941f9d9c5ecd9cf8ba7c87ea39ca6f30f2ccf100d WatchSource:0}: Error finding container 8ae1e077da559a0ce02c919941f9d9c5ecd9cf8ba7c87ea39ca6f30f2ccf100d: Status 404 returned error can't find the container with id 8ae1e077da559a0ce02c919941f9d9c5ecd9cf8ba7c87ea39ca6f30f2ccf100d
Nov 03 20:08:02.286334 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:02.286290    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:08:02.286424 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:02.286386    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:08:02.286490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:02.286451    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:08:03.227728 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:03.226905    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" event={"ID":"5789c64e-f731-4af7-96c0-f044fd64aaf6","Type":"ContainerStarted","Data":"5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca"}
Nov 03 20:08:03.227728 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:03.226975    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" event={"ID":"5789c64e-f731-4af7-96c0-f044fd64aaf6","Type":"ContainerStarted","Data":"8ae1e077da559a0ce02c919941f9d9c5ecd9cf8ba7c87ea39ca6f30f2ccf100d"}
Nov 03 20:08:03.301464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:03.297906    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" podStartSLOduration=2.2978859959999998 podStartE2EDuration="2.297885996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:03.25695386 +0000 UTC m=+1665.769775642" watchObservedRunningTime="2026-11-03 20:08:03.297885996 +0000 UTC m=+1665.810707782"
Nov 03 20:08:03.733245 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:03.733215    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:03.734881 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:03.734855    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:08:03.735117 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:03.735091    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:08:04.730761 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:04.730727    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:08:04.731266 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:04.730784    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:08:05.148471 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:05.147505    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:08:05.730716 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:05.730681    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:08:05.730907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:05.730747    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:08:06.731465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:06.731401    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:08:06.731892 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:06.731491    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:08:07.730346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:07.730311    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:08:07.730578 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:07.730362    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:08:08.731460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:08.731381    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:08:08.731460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:08.731452    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:08:09.735099 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:09.735065    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:09.735573 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:09.735353    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:09.736213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:09.736195    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:08:24.031789 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:24.031739    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"]
Nov 03 20:08:24.038425 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:24.038384    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9806786-z59mf"]
Nov 03 20:08:25.676679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:25.674810    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab39436f-a4a2-4b55-836e-d265ca01c0c1" path="/var/lib/kubelet/pods/ab39436f-a4a2-4b55-836e-d265ca01c0c1/volumes"
Nov 03 20:08:26.421643 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:26.421599    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-28w6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:08:26.421992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:26.421948    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:08:28.819451 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:08:28.819376    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:08:29.951446 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:29.951383    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"]
Nov 03 20:08:29.958870 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:29.958827    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:29.963741 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:29.963713    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-acd98d21-serving-cert\""
Nov 03 20:08:29.966478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:29.966410    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"]
Nov 03 20:08:30.027677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.027634    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0f3867-0e7f-4897-a8a7-652849d5799a-openshift-service-ca\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.027871 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.027840    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-maas-api-tls\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.027920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.027907    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-metrics-certs\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.027996 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.027961    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445pf\" (UniqueName: \"kubernetes.io/projected/2a0f3867-0e7f-4897-a8a7-652849d5799a-kube-api-access-445pf\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.145343 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.145313    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-metrics-certs\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.145779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.145755    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445pf\" (UniqueName: \"kubernetes.io/projected/2a0f3867-0e7f-4897-a8a7-652849d5799a-kube-api-access-445pf\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.146188 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.146152    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0f3867-0e7f-4897-a8a7-652849d5799a-openshift-service-ca\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.146346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.146322    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-maas-api-tls\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.147061 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:08:30.146528    5600 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-serving-cert: secret "maas-api-e2e-mt-acd98d21-serving-cert" not found
Nov 03 20:08:30.147283 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:08:30.147268    5600 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-maas-api-tls podName:2a0f3867-0e7f-4897-a8a7-652849d5799a nodeName:}" failed. No retries permitted until 2026-11-03 20:08:30.647243878 +0000 UTC m=+1693.160065655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-maas-api-tls") pod "maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6" (UID: "2a0f3867-0e7f-4897-a8a7-652849d5799a") : secret "maas-api-e2e-mt-acd98d21-serving-cert" not found
Nov 03 20:08:30.147799 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.147778    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0f3867-0e7f-4897-a8a7-652849d5799a-openshift-service-ca\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.151158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.151139    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-metrics-certs\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.171640 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.171606    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445pf\" (UniqueName: \"kubernetes.io/projected/2a0f3867-0e7f-4897-a8a7-652849d5799a-kube-api-access-445pf\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.658858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.658818    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-maas-api-tls\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.669102 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.669067    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-maas-api-tls\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:30.889826 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:30.889792    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:31.276117 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:31.276001    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"]
Nov 03 20:08:31.285929 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:08:31.285749    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0f3867_0e7f_4897_a8a7_652849d5799a.slice/crio-63ce59e5fea09b1d353504c986600aab74e11fa73454c7493c8284d9cff8fefc WatchSource:0}: Error finding container 63ce59e5fea09b1d353504c986600aab74e11fa73454c7493c8284d9cff8fefc: Status 404 returned error can't find the container with id 63ce59e5fea09b1d353504c986600aab74e11fa73454c7493c8284d9cff8fefc
Nov 03 20:08:31.296474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:31.293770    5600 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:08:31.675850 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:31.675809    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6" event={"ID":"2a0f3867-0e7f-4897-a8a7-652849d5799a","Type":"ContainerStarted","Data":"63ce59e5fea09b1d353504c986600aab74e11fa73454c7493c8284d9cff8fefc"}
Nov 03 20:08:32.688209 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:32.688170    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6" event={"ID":"2a0f3867-0e7f-4897-a8a7-652849d5799a","Type":"ContainerStarted","Data":"8fde46ca294c90691c4ffd440d41d7fc368b781b9c66075c186533e79407775a"}
Nov 03 20:08:32.688645 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:32.688297    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:32.717967 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:32.717905    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6" podStartSLOduration=3.095522566 podStartE2EDuration="3.717886396s" totalImagesPullingTime="622.36383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:32.71405907 +0000 UTC m=+1695.226880874" watchObservedRunningTime="2026-11-03 20:08:32.717886396 +0000 UTC m=+1695.230708178"
Nov 03 20:08:38.704312 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:38.704280    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:08:45.889127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.889084    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"]
Nov 03 20:08:45.894058 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.894026    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:45.902755 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.902715    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-acd98d21\"/\"kube-root-ca.crt\""
Nov 03 20:08:45.903033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.903013    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-acd98d21\"/\"openshift-service-ca.crt\""
Nov 03 20:08:45.903289 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.903271    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-acd98d21\"/\"e2e-lifecycle-model-acd98d21-kserve-self-signed-certs\""
Nov 03 20:08:45.903518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.903501    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-acd98d21\"/\"default-dockercfg-swggl\""
Nov 03 20:08:45.906929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.906905    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tmp-dir\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:45.907343 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.907239    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-home\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:45.907995 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.907967    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgj4c\" (UniqueName: \"kubernetes.io/projected/8316a979-44c7-4b82-8df4-1a1ced93d9a3-kube-api-access-sgj4c\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:45.908116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.908075    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tls-certs\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:45.908321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.908303    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-model-cache\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:45.908406 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.908354    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-dshm\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:45.915381 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:45.915356    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"]
Nov 03 20:08:46.010194 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.010158    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-model-cache\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.010397 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.010224    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-dshm\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.010397 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.010323    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tmp-dir\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.010397 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.010366    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-home\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.010624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.010402    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgj4c\" (UniqueName: \"kubernetes.io/projected/8316a979-44c7-4b82-8df4-1a1ced93d9a3-kube-api-access-sgj4c\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.010624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.010534    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tls-certs\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.010767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.010659    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-model-cache\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.010906 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.010881    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-home\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.010974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.010908    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tmp-dir\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.013372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.013345    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tls-certs\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.013550 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.013396    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-dshm\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.025721 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.025696    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgj4c\" (UniqueName: \"kubernetes.io/projected/8316a979-44c7-4b82-8df4-1a1ced93d9a3-kube-api-access-sgj4c\") pod \"e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") " pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.237168 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.235719    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:46.761831 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:08:46.761736    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8316a979_44c7_4b82_8df4_1a1ced93d9a3.slice/crio-21c497e3abae61ef1df170e65462ca61d8aa42d032cfb6546af8d4cca1638d7b WatchSource:0}: Error finding container 21c497e3abae61ef1df170e65462ca61d8aa42d032cfb6546af8d4cca1638d7b: Status 404 returned error can't find the container with id 21c497e3abae61ef1df170e65462ca61d8aa42d032cfb6546af8d4cca1638d7b
Nov 03 20:08:46.773180 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.773152    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"]
Nov 03 20:08:46.845630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:46.845466    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j" event={"ID":"8316a979-44c7-4b82-8df4-1a1ced93d9a3","Type":"ContainerStarted","Data":"21c497e3abae61ef1df170e65462ca61d8aa42d032cfb6546af8d4cca1638d7b"}
Nov 03 20:08:47.856515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:47.856477    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j" event={"ID":"8316a979-44c7-4b82-8df4-1a1ced93d9a3","Type":"ContainerStarted","Data":"3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c"}
Nov 03 20:08:47.874603 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:47.874556    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j" podStartSLOduration=2.874540276 podStartE2EDuration="2.874540276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:47.873249433 +0000 UTC m=+1710.386071215" watchObservedRunningTime="2026-11-03 20:08:47.874540276 +0000 UTC m=+1710.387362057"
Nov 03 20:08:48.236279 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:48.236223    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:56.244456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:56.241972    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:56.272849 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:56.272817    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:57.015480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:08:57.013313    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:08:58.799651 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:08:58.799612    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:09:05.621668 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:05.621623    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:09:05.653803 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:05.653760    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:09:28.803774 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:09:28.803718    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:09:42.625406 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.625370    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"]
Nov 03 20:09:42.648997 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.648959    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.652970 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.652944    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1962f9-serving-cert\""
Nov 03 20:09:42.660251 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.660217    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"]
Nov 03 20:09:42.761470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.747461    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-metrics-certs\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.761937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.761904    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31343dd5-bf61-42bc-9b78-3fcf989cad5b-openshift-service-ca\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.762171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.762155    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-maas-api-tls\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.762639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.762399    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8hb\" (UniqueName: \"kubernetes.io/projected/31343dd5-bf61-42bc-9b78-3fcf989cad5b-kube-api-access-7g8hb\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.864376 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.864340    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-metrics-certs\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.864573 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.864446    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31343dd5-bf61-42bc-9b78-3fcf989cad5b-openshift-service-ca\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.864573 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.864511    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-maas-api-tls\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.864573 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.864555    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8hb\" (UniqueName: \"kubernetes.io/projected/31343dd5-bf61-42bc-9b78-3fcf989cad5b-kube-api-access-7g8hb\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.865269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.865244    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31343dd5-bf61-42bc-9b78-3fcf989cad5b-openshift-service-ca\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.867022 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.866999    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-metrics-certs\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.867126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.867113    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-maas-api-tls\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.872900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.872876    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8hb\" (UniqueName: \"kubernetes.io/projected/31343dd5-bf61-42bc-9b78-3fcf989cad5b-kube-api-access-7g8hb\") pod \"maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:42.968749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:42.968711    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:43.455796 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:43.442579    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"]
Nov 03 20:09:43.924499 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:43.914128    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd" event={"ID":"31343dd5-bf61-42bc-9b78-3fcf989cad5b","Type":"ContainerStarted","Data":"dc6c87b5b0a2ed7b114eeee655eba561c2832b7b74a258cd9e9d561b9b7bf1c1"}
Nov 03 20:09:44.926091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:44.926053    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd" event={"ID":"31343dd5-bf61-42bc-9b78-3fcf989cad5b","Type":"ContainerStarted","Data":"5933ee95887b4fed2cade3445f6adfe3cff167436ddf27f10fb31205784264b3"}
Nov 03 20:09:44.926589 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:44.926156    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:44.945526 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:44.945470    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd" podStartSLOduration=2.372355438 podStartE2EDuration="2.945456628s" totalImagesPullingTime="573.10119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:44.943191214 +0000 UTC m=+1767.456013011" watchObservedRunningTime="2026-11-03 20:09:44.945456628 +0000 UTC m=+1767.458278431"
Nov 03 20:09:49.048188 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.048145    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"]
Nov 03 20:09:49.061294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.056044    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.062497 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.062473    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-cad531-serving-cert\""
Nov 03 20:09:49.062730 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.062709    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"]
Nov 03 20:09:49.105239 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.105185    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.105530 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.105506    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.105800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.105779    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrgg\" (UniqueName: \"kubernetes.io/projected/fcfea225-609d-4768-9ec1-b9dcf38c350f-kube-api-access-pqrgg\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.105893 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.105854    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfea225-609d-4768-9ec1-b9dcf38c350f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.208953 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.208488    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrgg\" (UniqueName: \"kubernetes.io/projected/fcfea225-609d-4768-9ec1-b9dcf38c350f-kube-api-access-pqrgg\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.208953 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.208607    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfea225-609d-4768-9ec1-b9dcf38c350f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.208953 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.208817    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.208953 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.208845    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.263578 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.258359    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrgg\" (UniqueName: \"kubernetes.io/projected/fcfea225-609d-4768-9ec1-b9dcf38c350f-kube-api-access-pqrgg\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.263578 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.258823    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfea225-609d-4768-9ec1-b9dcf38c350f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.263578 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.259296    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.325675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.325626    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.381588 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.381539    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:49.854694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.854664    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"]
Nov 03 20:09:49.970363 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:49.970326    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r" event={"ID":"fcfea225-609d-4768-9ec1-b9dcf38c350f","Type":"ContainerStarted","Data":"d0167fcd36d5b6f53baa91d1ff9c9536dc0bca99d625a77822ee6e18a89ff875"}
Nov 03 20:09:50.814538 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:50.812354    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"]
Nov 03 20:09:50.819240 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:50.818240    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:50.825561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:50.825534    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"]
Nov 03 20:09:50.831942 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:50.830659    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b9dc48-serving-cert\""
Nov 03 20:09:50.933589 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:50.933559    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-maas-api-tls\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:50.933589 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:50.933591    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae5f00e-8af6-410b-b148-a08cb888051b-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:50.933847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:50.933614    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbmvp\" (UniqueName: \"kubernetes.io/projected/fae5f00e-8af6-410b-b148-a08cb888051b-kube-api-access-bbmvp\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:50.934037 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:50.934013    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-metrics-certs\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:50.940173 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:50.940150    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:09:50.980125 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:50.980084    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r" event={"ID":"fcfea225-609d-4768-9ec1-b9dcf38c350f","Type":"ContainerStarted","Data":"cc4ce8a7be4f66116cbd94504d86d20757977f21c855e60be61cbc03be8574c3"}
Nov 03 20:09:50.980319 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:50.980191    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:51.003150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.003089    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r" podStartSLOduration=1.451778092 podStartE2EDuration="2.003070751s" totalImagesPullingTime="551.292659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:51.001681365 +0000 UTC m=+1773.514503147" watchObservedRunningTime="2026-11-03 20:09:51.003070751 +0000 UTC m=+1773.515892534"
Nov 03 20:09:51.036395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.036064    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-metrics-certs\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:51.036395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.036248    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-maas-api-tls\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:51.036395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.036295    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae5f00e-8af6-410b-b148-a08cb888051b-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:51.036395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.036330    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmvp\" (UniqueName: \"kubernetes.io/projected/fae5f00e-8af6-410b-b148-a08cb888051b-kube-api-access-bbmvp\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:51.037243 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.037166    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae5f00e-8af6-410b-b148-a08cb888051b-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:51.039087 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.039058    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-maas-api-tls\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:51.039087 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.039082    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-metrics-certs\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:51.044502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.044417    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmvp\" (UniqueName: \"kubernetes.io/projected/fae5f00e-8af6-410b-b148-a08cb888051b-kube-api-access-bbmvp\") pod \"maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:51.152395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.152121    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:51.688459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.686041    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"]
Nov 03 20:09:51.996678 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:51.996568    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws" event={"ID":"fae5f00e-8af6-410b-b148-a08cb888051b","Type":"ContainerStarted","Data":"0d803cb2029ff442a2d0098e8686fe6e0e99af44b4e783eebc3a1408c09f505c"}
Nov 03 20:09:53.007500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:53.007467    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws" event={"ID":"fae5f00e-8af6-410b-b148-a08cb888051b","Type":"ContainerStarted","Data":"29e22e5844236cc8aba5ba78121dd4a682deab4017c16d1652931f667a087288"}
Nov 03 20:09:53.007995 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:53.007535    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:09:53.032021 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:53.031960    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws" podStartSLOduration=2.568071272 podStartE2EDuration="3.031945274s" totalImagesPullingTime="463.874002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:53.027297004 +0000 UTC m=+1775.540118786" watchObservedRunningTime="2026-11-03 20:09:53.031945274 +0000 UTC m=+1775.544767055"
Nov 03 20:09:57.002919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:57.002892    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:09:58.812982 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:09:58.812945    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:09:59.025160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:09:59.025128    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:10:19.170719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:19.170684    5600 scope.go:122] "RemoveContainer" containerID="20b1c43f03132fdd210bf629d794eb5322bf045cc32de3ff4e5532ec4ec10025"
Nov 03 20:10:22.341528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.341492    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"]
Nov 03 20:10:22.351973 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.351943    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.356017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.355988    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cd7f5c-serving-cert\""
Nov 03 20:10:22.356528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.356498    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"]
Nov 03 20:10:22.460814 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.460780    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-maas-api-tls\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.461053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.461035    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f6045e-e98a-4f1e-8208-cff2804bd400-openshift-service-ca\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.461107 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.461081    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fltbg\" (UniqueName: \"kubernetes.io/projected/84f6045e-e98a-4f1e-8208-cff2804bd400-kube-api-access-fltbg\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.461229 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.461209    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-metrics-certs\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.563534 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.563498    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-maas-api-tls\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.563747 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.563720    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f6045e-e98a-4f1e-8208-cff2804bd400-openshift-service-ca\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.563811 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.563766    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fltbg\" (UniqueName: \"kubernetes.io/projected/84f6045e-e98a-4f1e-8208-cff2804bd400-kube-api-access-fltbg\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.563860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.563827    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-metrics-certs\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.564364 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.564338    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f6045e-e98a-4f1e-8208-cff2804bd400-openshift-service-ca\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.566662 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.566641    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-metrics-certs\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.566775 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.566719    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-maas-api-tls\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.571772 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.571747    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fltbg\" (UniqueName: \"kubernetes.io/projected/84f6045e-e98a-4f1e-8208-cff2804bd400-kube-api-access-fltbg\") pod \"maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:22.702909 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:22.702881    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:23.231776 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:23.231653    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"]
Nov 03 20:10:23.480517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:23.479424    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n" event={"ID":"84f6045e-e98a-4f1e-8208-cff2804bd400","Type":"ContainerStarted","Data":"60308c363af7bb9c024e15f8fd1391baa78f3d0f31f0d2bba99f159c30401ead"}
Nov 03 20:10:24.488772 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:24.488732    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n" event={"ID":"84f6045e-e98a-4f1e-8208-cff2804bd400","Type":"ContainerStarted","Data":"d1f88939a6af4ce62c32c66efebd97675ea86b3baff682e1b67ff11d0278e6ed"}
Nov 03 20:10:24.489328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:24.488826    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:24.509183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:24.509123    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n" podStartSLOduration=2.076010497 podStartE2EDuration="2.509109111s" totalImagesPullingTime="433.098614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:24.504962487 +0000 UTC m=+1807.017784270" watchObservedRunningTime="2026-11-03 20:10:24.509109111 +0000 UTC m=+1807.021930893"
Nov 03 20:10:26.942138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:26.942047    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11044-6b38a6"]
Nov 03 20:10:26.947052 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:26.947027    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11044-6b38a6"
Nov 03 20:10:26.950691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:26.950664    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4mfkw\""
Nov 03 20:10:26.950962 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:26.950941    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11044-6b38a6"]
Nov 03 20:10:27.069339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:27.069305    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4j6\" (UniqueName: \"kubernetes.io/projected/fc2b1db6-58af-45ad-a6a5-5cb1f6559947-kube-api-access-hn4j6\") pod \"test-curl-11044-6b38a6\" (UID: \"fc2b1db6-58af-45ad-a6a5-5cb1f6559947\") " pod="opendatahub/test-curl-11044-6b38a6"
Nov 03 20:10:27.171948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:27.171909    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4j6\" (UniqueName: \"kubernetes.io/projected/fc2b1db6-58af-45ad-a6a5-5cb1f6559947-kube-api-access-hn4j6\") pod \"test-curl-11044-6b38a6\" (UID: \"fc2b1db6-58af-45ad-a6a5-5cb1f6559947\") " pod="opendatahub/test-curl-11044-6b38a6"
Nov 03 20:10:27.180581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:27.180550    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4j6\" (UniqueName: \"kubernetes.io/projected/fc2b1db6-58af-45ad-a6a5-5cb1f6559947-kube-api-access-hn4j6\") pod \"test-curl-11044-6b38a6\" (UID: \"fc2b1db6-58af-45ad-a6a5-5cb1f6559947\") " pod="opendatahub/test-curl-11044-6b38a6"
Nov 03 20:10:27.272721 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:27.272630    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11044-6b38a6"
Nov 03 20:10:27.563063 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:27.563033    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11044-6b38a6"]
Nov 03 20:10:27.563376 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:10:27.563349    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc2b1db6_58af_45ad_a6a5_5cb1f6559947.slice/crio-95e31e376cbe36fa0076573fff084df31b76c7c23978af02d7cb78efb8865af3 WatchSource:0}: Error finding container 95e31e376cbe36fa0076573fff084df31b76c7c23978af02d7cb78efb8865af3: Status 404 returned error can't find the container with id 95e31e376cbe36fa0076573fff084df31b76c7c23978af02d7cb78efb8865af3
Nov 03 20:10:28.531822 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:28.531776    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11044-6b38a6" event={"ID":"fc2b1db6-58af-45ad-a6a5-5cb1f6559947","Type":"ContainerStarted","Data":"a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa"}
Nov 03 20:10:28.532329 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:28.531833    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11044-6b38a6" event={"ID":"fc2b1db6-58af-45ad-a6a5-5cb1f6559947","Type":"ContainerStarted","Data":"95e31e376cbe36fa0076573fff084df31b76c7c23978af02d7cb78efb8865af3"}
Nov 03 20:10:28.551483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:28.551415    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11044-6b38a6" podStartSLOduration=2.551402573 podStartE2EDuration="2.551402573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:28.546490806 +0000 UTC m=+1811.059312589" watchObservedRunningTime="2026-11-03 20:10:28.551402573 +0000 UTC m=+1811.064224355"
Nov 03 20:10:28.817269 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:10:28.817164    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:10:29.352047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:29.352014    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11044-6b38a6"]
Nov 03 20:10:29.368856 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:29.368827    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11044-6b38a6"]
Nov 03 20:10:29.848184 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:29.843790    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"]
Nov 03 20:10:29.848184 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:29.844761    5600 cpu_manager.go:414] "containerMap: removing container" podUID="fc2b1db6-58af-45ad-a6a5-5cb1f6559947" containerName="test-curl-11044-6b38a6"
Nov 03 20:10:29.848184 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:29.844779    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2b1db6-58af-45ad-a6a5-5cb1f6559947" containerName="test-curl-11044-6b38a6"
Nov 03 20:10:29.848184 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:29.844999    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2b1db6-58af-45ad-a6a5-5cb1f6559947" containerName="test-curl-11044-6b38a6"
Nov 03 20:10:29.849555 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:29.849532    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:29.860726 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:29.860700    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-a476f8-serving-cert\""
Nov 03 20:10:29.874493 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:29.874463    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"]
Nov 03 20:10:30.033725 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.033679    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.033943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.033747    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbj8k\" (UniqueName: \"kubernetes.io/projected/e0121c77-f76c-4041-8dc3-707f115f5bcb-kube-api-access-rbj8k\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.034214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.034162    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0121c77-f76c-4041-8dc3-707f115f5bcb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.034325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.034222    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.137493 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.137464    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0121c77-f76c-4041-8dc3-707f115f5bcb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.137679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.137502    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.137722 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.137704    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.138072 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.138048    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbj8k\" (UniqueName: \"kubernetes.io/projected/e0121c77-f76c-4041-8dc3-707f115f5bcb-kube-api-access-rbj8k\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.138557 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.138529    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0121c77-f76c-4041-8dc3-707f115f5bcb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.141006 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.140982    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.143241 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.143216    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.147690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.147666    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbj8k\" (UniqueName: \"kubernetes.io/projected/e0121c77-f76c-4041-8dc3-707f115f5bcb-kube-api-access-rbj8k\") pod \"maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.173963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.173931    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:30.474690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.474660    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"]
Nov 03 20:10:30.478501 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:10:30.478471    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0121c77_f76c_4041_8dc3_707f115f5bcb.slice/crio-6d31c60046043b2237269f4452fdcb5a4417fd2b143d5cd60db91004c3f6a17b WatchSource:0}: Error finding container 6d31c60046043b2237269f4452fdcb5a4417fd2b143d5cd60db91004c3f6a17b: Status 404 returned error can't find the container with id 6d31c60046043b2237269f4452fdcb5a4417fd2b143d5cd60db91004c3f6a17b
Nov 03 20:10:30.504543 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.504517    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:10:30.563167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.563125    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11044-6b38a6" podUID="fc2b1db6-58af-45ad-a6a5-5cb1f6559947" containerName="test-curl-11044-6b38a6" containerID="cri-o://a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa" gracePeriod=2
Nov 03 20:10:30.563513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:30.563488    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2" event={"ID":"e0121c77-f76c-4041-8dc3-707f115f5bcb","Type":"ContainerStarted","Data":"6d31c60046043b2237269f4452fdcb5a4417fd2b143d5cd60db91004c3f6a17b"}
Nov 03 20:10:31.574672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:31.574555    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2" event={"ID":"e0121c77-f76c-4041-8dc3-707f115f5bcb","Type":"ContainerStarted","Data":"bddf5e8552bcc3c12b0a21af5a97ee23d9954aa1e2ed74467147e6714cdcf653"}
Nov 03 20:10:31.575182 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:31.574693    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:31.598032 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:31.597974    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2" podStartSLOduration=2.090536641 podStartE2EDuration="2.597955997s" totalImagesPullingTime="507.419356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:31.592965 +0000 UTC m=+1814.105786804" watchObservedRunningTime="2026-11-03 20:10:31.597955997 +0000 UTC m=+1814.110777780"
Nov 03 20:10:32.839001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:32.838933    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11044-6b38a6"
Nov 03 20:10:32.949714 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:32.949682    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc2b1db6-58af-45ad-a6a5-5cb1f6559947-kube-api-access-hn4j6\" (UniqueName: \"kubernetes.io/projected/fc2b1db6-58af-45ad-a6a5-5cb1f6559947-kube-api-access-hn4j6\") pod \"fc2b1db6-58af-45ad-a6a5-5cb1f6559947\" (UID: \"fc2b1db6-58af-45ad-a6a5-5cb1f6559947\") "
Nov 03 20:10:32.952607 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:32.952575    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2b1db6-58af-45ad-a6a5-5cb1f6559947-kube-api-access-hn4j6" pod "fc2b1db6-58af-45ad-a6a5-5cb1f6559947" (UID: "fc2b1db6-58af-45ad-a6a5-5cb1f6559947"). InnerVolumeSpecName "kube-api-access-hn4j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:33.051794 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:33.051760    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn4j6\" (UniqueName: \"kubernetes.io/projected/fc2b1db6-58af-45ad-a6a5-5cb1f6559947-kube-api-access-hn4j6\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:33.598649 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:33.598613    5600 generic.go:356] "Generic (PLEG): container finished" podID="fc2b1db6-58af-45ad-a6a5-5cb1f6559947" containerID="a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa" exitCode=137
Nov 03 20:10:33.598649 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:33.598657    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11044-6b38a6"
Nov 03 20:10:33.599269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:33.598710    5600 scope.go:122] "RemoveContainer" containerID="a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa"
Nov 03 20:10:33.614379 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:10:33.614340    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11044-6b38a6\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="fc2b1db6-58af-45ad-a6a5-5cb1f6559947" pod="opendatahub/test-curl-11044-6b38a6"
Nov 03 20:10:33.619560 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:33.619534    5600 scope.go:122] "RemoveContainer" containerID="a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa"
Nov 03 20:10:33.619895 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:10:33.619867    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa\": container with ID starting with a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa not found: ID does not exist" containerID="a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa"
Nov 03 20:10:33.620011 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:33.619903    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa"} err="failed to get container status \"a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa\": rpc error: code = NotFound desc = could not find container \"a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa\": container with ID starting with a8fb667de2fe4de4cb35904f7dd530dc9097cc831908de24accdb805626cbcfa not found: ID does not exist"
Nov 03 20:10:33.624926 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:33.624899    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc2b1db6-58af-45ad-a6a5-5cb1f6559947" path="/var/lib/kubelet/pods/fc2b1db6-58af-45ad-a6a5-5cb1f6559947/volumes"
Nov 03 20:10:37.666490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:37.661666    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:10:53.544616 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:53.544583    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11041-71b975"]
Nov 03 20:10:53.549284 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:53.549257    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-71b975"
Nov 03 20:10:53.557399 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:53.557370    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11041-71b975"]
Nov 03 20:10:53.568872 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:53.568846    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4mfkw\""
Nov 03 20:10:53.660031 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:53.659526    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b9mh\" (UniqueName: \"kubernetes.io/projected/f3ce976f-ccb2-4a64-8250-701b3f634230-kube-api-access-2b9mh\") pod \"test-curl-11041-71b975\" (UID: \"f3ce976f-ccb2-4a64-8250-701b3f634230\") " pod="opendatahub/test-curl-11041-71b975"
Nov 03 20:10:53.762146 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:53.762113    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9mh\" (UniqueName: \"kubernetes.io/projected/f3ce976f-ccb2-4a64-8250-701b3f634230-kube-api-access-2b9mh\") pod \"test-curl-11041-71b975\" (UID: \"f3ce976f-ccb2-4a64-8250-701b3f634230\") " pod="opendatahub/test-curl-11041-71b975"
Nov 03 20:10:53.771534 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:53.771503    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9mh\" (UniqueName: \"kubernetes.io/projected/f3ce976f-ccb2-4a64-8250-701b3f634230-kube-api-access-2b9mh\") pod \"test-curl-11041-71b975\" (UID: \"f3ce976f-ccb2-4a64-8250-701b3f634230\") " pod="opendatahub/test-curl-11041-71b975"
Nov 03 20:10:53.875853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:53.875773    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-71b975"
Nov 03 20:10:54.166639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:54.166605    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11041-71b975"]
Nov 03 20:10:54.167039 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:10:54.167010    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3ce976f_ccb2_4a64_8250_701b3f634230.slice/crio-ab4b918b525c593ae98354a4961d6f37ecf1569455e199f4ac025a15e123d718 WatchSource:0}: Error finding container ab4b918b525c593ae98354a4961d6f37ecf1569455e199f4ac025a15e123d718: Status 404 returned error can't find the container with id ab4b918b525c593ae98354a4961d6f37ecf1569455e199f4ac025a15e123d718
Nov 03 20:10:54.912053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:54.912014    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11041-71b975" event={"ID":"f3ce976f-ccb2-4a64-8250-701b3f634230","Type":"ContainerStarted","Data":"2d8b569a948104800f5bb7a71d21b01bf18945e4ecd7a7aca52c60608fa05bec"}
Nov 03 20:10:54.914124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:54.914096    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11041-71b975" event={"ID":"f3ce976f-ccb2-4a64-8250-701b3f634230","Type":"ContainerStarted","Data":"ab4b918b525c593ae98354a4961d6f37ecf1569455e199f4ac025a15e123d718"}
Nov 03 20:10:54.938813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:54.938758    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11041-71b975" podStartSLOduration=1.938740457 podStartE2EDuration="1.938740457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:54.930665072 +0000 UTC m=+1837.443486855" watchObservedRunningTime="2026-11-03 20:10:54.938740457 +0000 UTC m=+1837.451562239"
Nov 03 20:10:55.924041 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:55.923756    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11041-71b975"]
Nov 03 20:10:55.931228 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:55.931194    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11041-71b975"]
Nov 03 20:10:56.079082 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:56.079042    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11041-80ea48"]
Nov 03 20:10:56.079789 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:56.079772    5600 cpu_manager.go:414] "containerMap: removing container" podUID="f3ce976f-ccb2-4a64-8250-701b3f634230" containerName="test-curl-11041-71b975"
Nov 03 20:10:56.079839 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:56.079791    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ce976f-ccb2-4a64-8250-701b3f634230" containerName="test-curl-11041-71b975"
Nov 03 20:10:56.079910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:56.079901    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ce976f-ccb2-4a64-8250-701b3f634230" containerName="test-curl-11041-71b975"
Nov 03 20:10:56.083834 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:56.083810    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-80ea48"
Nov 03 20:10:56.093762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:56.093726    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11041-80ea48"]
Nov 03 20:10:56.230369 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:56.230268    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcj8q\" (UniqueName: \"kubernetes.io/projected/bc1360c5-2ba3-476b-8839-18e6e22c1683-kube-api-access-fcj8q\") pod \"test-curl-11041-80ea48\" (UID: \"bc1360c5-2ba3-476b-8839-18e6e22c1683\") " pod="opendatahub/test-curl-11041-80ea48"
Nov 03 20:10:56.332876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:56.332832    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcj8q\" (UniqueName: \"kubernetes.io/projected/bc1360c5-2ba3-476b-8839-18e6e22c1683-kube-api-access-fcj8q\") pod \"test-curl-11041-80ea48\" (UID: \"bc1360c5-2ba3-476b-8839-18e6e22c1683\") " pod="opendatahub/test-curl-11041-80ea48"
Nov 03 20:10:56.341211 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:56.341181    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcj8q\" (UniqueName: \"kubernetes.io/projected/bc1360c5-2ba3-476b-8839-18e6e22c1683-kube-api-access-fcj8q\") pod \"test-curl-11041-80ea48\" (UID: \"bc1360c5-2ba3-476b-8839-18e6e22c1683\") " pod="opendatahub/test-curl-11041-80ea48"
Nov 03 20:10:56.435493 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:56.435459    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-80ea48"
Nov 03 20:10:56.843475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:56.837238    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11041-80ea48"]
Nov 03 20:10:57.018067 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:57.018027    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11041-80ea48" event={"ID":"bc1360c5-2ba3-476b-8839-18e6e22c1683","Type":"ContainerStarted","Data":"29759d3de7becdbc7ae12d8389fc10551006a024a2f6a796ad368a5766738b05"}
Nov 03 20:10:58.029356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:58.029325    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11041-80ea48" event={"ID":"bc1360c5-2ba3-476b-8839-18e6e22c1683","Type":"ContainerStarted","Data":"4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7"}
Nov 03 20:10:58.029823 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:58.029602    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11041-71b975" podUID="f3ce976f-ccb2-4a64-8250-701b3f634230" containerName="test-curl-11041-71b975" containerID="cri-o://2d8b569a948104800f5bb7a71d21b01bf18945e4ecd7a7aca52c60608fa05bec" gracePeriod=2
Nov 03 20:10:58.046661 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:58.046609    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11041-80ea48" podStartSLOduration=2.046594815 podStartE2EDuration="2.046594815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:58.042627289 +0000 UTC m=+1840.555449072" watchObservedRunningTime="2026-11-03 20:10:58.046594815 +0000 UTC m=+1840.559416596"
Nov 03 20:10:58.751401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:58.751360    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11041-80ea48"]
Nov 03 20:10:58.760710 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:58.760682    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11041-80ea48"]
Nov 03 20:10:58.814635 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:10:58.814595    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:10:58.967471 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:58.966850    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11041-9c3805"]
Nov 03 20:10:58.997458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:58.992699    5600 cpu_manager.go:414] "containerMap: removing container" podUID="bc1360c5-2ba3-476b-8839-18e6e22c1683" containerName="test-curl-11041-80ea48"
Nov 03 20:10:58.997458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:58.992737    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1360c5-2ba3-476b-8839-18e6e22c1683" containerName="test-curl-11041-80ea48"
Nov 03 20:10:58.997458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:58.993138    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1360c5-2ba3-476b-8839-18e6e22c1683" containerName="test-curl-11041-80ea48"
Nov 03 20:10:59.043614 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:59.039782    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-9c3805"
Nov 03 20:10:59.079877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:59.079848    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11041-9c3805"]
Nov 03 20:10:59.250766 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:59.250722    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjfxr\" (UniqueName: \"kubernetes.io/projected/ccd2ff8f-eb3d-4b89-9a36-c1860b271d55-kube-api-access-pjfxr\") pod \"test-curl-11041-9c3805\" (UID: \"ccd2ff8f-eb3d-4b89-9a36-c1860b271d55\") " pod="opendatahub/test-curl-11041-9c3805"
Nov 03 20:10:59.352374 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:59.352266    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjfxr\" (UniqueName: \"kubernetes.io/projected/ccd2ff8f-eb3d-4b89-9a36-c1860b271d55-kube-api-access-pjfxr\") pod \"test-curl-11041-9c3805\" (UID: \"ccd2ff8f-eb3d-4b89-9a36-c1860b271d55\") " pod="opendatahub/test-curl-11041-9c3805"
Nov 03 20:10:59.367411 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:59.367373    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjfxr\" (UniqueName: \"kubernetes.io/projected/ccd2ff8f-eb3d-4b89-9a36-c1860b271d55-kube-api-access-pjfxr\") pod \"test-curl-11041-9c3805\" (UID: \"ccd2ff8f-eb3d-4b89-9a36-c1860b271d55\") " pod="opendatahub/test-curl-11041-9c3805"
Nov 03 20:10:59.377911 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:59.377879    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-9c3805"
Nov 03 20:10:59.677676 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:10:59.677649    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11041-9c3805"]
Nov 03 20:10:59.679288 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:10:59.679261    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccd2ff8f_eb3d_4b89_9a36_c1860b271d55.slice/crio-b15ff91b6dff975ac23ce51f2155be0d855b822d138e512a7804a50587f8161d WatchSource:0}: Error finding container b15ff91b6dff975ac23ce51f2155be0d855b822d138e512a7804a50587f8161d: Status 404 returned error can't find the container with id b15ff91b6dff975ac23ce51f2155be0d855b822d138e512a7804a50587f8161d
Nov 03 20:11:00.114815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:00.114778    5600 generic.go:356] "Generic (PLEG): container finished" podID="f3ce976f-ccb2-4a64-8250-701b3f634230" containerID="2d8b569a948104800f5bb7a71d21b01bf18945e4ecd7a7aca52c60608fa05bec" exitCode=137
Nov 03 20:11:00.117605 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:00.117547    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11041-80ea48" podUID="bc1360c5-2ba3-476b-8839-18e6e22c1683" containerName="test-curl-11041-80ea48" containerID="cri-o://4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7" gracePeriod=2
Nov 03 20:11:00.117798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:00.117680    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11041-9c3805" event={"ID":"ccd2ff8f-eb3d-4b89-9a36-c1860b271d55","Type":"ContainerStarted","Data":"3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919"}
Nov 03 20:11:00.117798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:00.117714    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11041-9c3805" event={"ID":"ccd2ff8f-eb3d-4b89-9a36-c1860b271d55","Type":"ContainerStarted","Data":"b15ff91b6dff975ac23ce51f2155be0d855b822d138e512a7804a50587f8161d"}
Nov 03 20:11:00.150064 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:00.150010    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11041-9c3805" podStartSLOduration=2.149991032 podStartE2EDuration="2.149991032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:00.146283292 +0000 UTC m=+1842.659105070" watchObservedRunningTime="2026-11-03 20:11:00.149991032 +0000 UTC m=+1842.662812815"
Nov 03 20:11:00.336468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:00.336374    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-71b975"
Nov 03 20:11:00.490570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:00.489551    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ce976f-ccb2-4a64-8250-701b3f634230-kube-api-access-2b9mh\" (UniqueName: \"kubernetes.io/projected/f3ce976f-ccb2-4a64-8250-701b3f634230-kube-api-access-2b9mh\") pod \"f3ce976f-ccb2-4a64-8250-701b3f634230\" (UID: \"f3ce976f-ccb2-4a64-8250-701b3f634230\") "
Nov 03 20:11:00.679539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:00.679005    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ce976f-ccb2-4a64-8250-701b3f634230-kube-api-access-2b9mh" pod "f3ce976f-ccb2-4a64-8250-701b3f634230" (UID: "f3ce976f-ccb2-4a64-8250-701b3f634230"). InnerVolumeSpecName "kube-api-access-2b9mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:00.694608 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:00.694567    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b9mh\" (UniqueName: \"kubernetes.io/projected/f3ce976f-ccb2-4a64-8250-701b3f634230-kube-api-access-2b9mh\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:01.097626 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.095374    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11041-9c3805"]
Nov 03 20:11:01.103299 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.103265    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11041-9c3805"]
Nov 03 20:11:01.133136 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.132126    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-71b975"
Nov 03 20:11:01.133136 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.132174    5600 scope.go:122] "RemoveContainer" containerID="2d8b569a948104800f5bb7a71d21b01bf18945e4ecd7a7aca52c60608fa05bec"
Nov 03 20:11:01.146032 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:01.146001    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11041-71b975\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="f3ce976f-ccb2-4a64-8250-701b3f634230" pod="opendatahub/test-curl-11041-71b975"
Nov 03 20:11:01.250774 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.250741    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11041-3417e9"]
Nov 03 20:11:01.252027 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.252001    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ccd2ff8f-eb3d-4b89-9a36-c1860b271d55" containerName="test-curl-11041-9c3805"
Nov 03 20:11:01.252174 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.252161    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd2ff8f-eb3d-4b89-9a36-c1860b271d55" containerName="test-curl-11041-9c3805"
Nov 03 20:11:01.252427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.252416    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd2ff8f-eb3d-4b89-9a36-c1860b271d55" containerName="test-curl-11041-9c3805"
Nov 03 20:11:01.266235 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.266193    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11041-3417e9"]
Nov 03 20:11:01.266628 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.266599    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-3417e9"
Nov 03 20:11:01.291799 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:01.291757    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11041-71b975\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="f3ce976f-ccb2-4a64-8250-701b3f634230" pod="opendatahub/test-curl-11041-71b975"
Nov 03 20:11:01.307763 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.307729    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8fzv\" (UniqueName: \"kubernetes.io/projected/0e442543-de64-4941-a3c8-c13e2183fa27-kube-api-access-p8fzv\") pod \"test-curl-11041-3417e9\" (UID: \"0e442543-de64-4941-a3c8-c13e2183fa27\") " pod="opendatahub/test-curl-11041-3417e9"
Nov 03 20:11:01.410325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.410283    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fzv\" (UniqueName: \"kubernetes.io/projected/0e442543-de64-4941-a3c8-c13e2183fa27-kube-api-access-p8fzv\") pod \"test-curl-11041-3417e9\" (UID: \"0e442543-de64-4941-a3c8-c13e2183fa27\") " pod="opendatahub/test-curl-11041-3417e9"
Nov 03 20:11:01.423408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.421365    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8fzv\" (UniqueName: \"kubernetes.io/projected/0e442543-de64-4941-a3c8-c13e2183fa27-kube-api-access-p8fzv\") pod \"test-curl-11041-3417e9\" (UID: \"0e442543-de64-4941-a3c8-c13e2183fa27\") " pod="opendatahub/test-curl-11041-3417e9"
Nov 03 20:11:01.585845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.585811    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-3417e9"
Nov 03 20:11:01.623579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.623546    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3ce976f-ccb2-4a64-8250-701b3f634230" path="/var/lib/kubelet/pods/f3ce976f-ccb2-4a64-8250-701b3f634230/volumes"
Nov 03 20:11:01.877212 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:11:01.877179    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e442543_de64_4941_a3c8_c13e2183fa27.slice/crio-dc08a876ca0e380656e7dcbb9186822d9cd1fa0dc75f478ea8ce96b3bde90eaf WatchSource:0}: Error finding container dc08a876ca0e380656e7dcbb9186822d9cd1fa0dc75f478ea8ce96b3bde90eaf: Status 404 returned error can't find the container with id dc08a876ca0e380656e7dcbb9186822d9cd1fa0dc75f478ea8ce96b3bde90eaf
Nov 03 20:11:01.877579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:01.877558    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11041-3417e9"]
Nov 03 20:11:02.166855 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:02.166816    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11041-3417e9" event={"ID":"0e442543-de64-4941-a3c8-c13e2183fa27","Type":"ContainerStarted","Data":"e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c"}
Nov 03 20:11:02.166855 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:02.166858    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11041-3417e9" event={"ID":"0e442543-de64-4941-a3c8-c13e2183fa27","Type":"ContainerStarted","Data":"dc08a876ca0e380656e7dcbb9186822d9cd1fa0dc75f478ea8ce96b3bde90eaf"}
Nov 03 20:11:02.167351 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:02.167072    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11041-9c3805" podUID="ccd2ff8f-eb3d-4b89-9a36-c1860b271d55" containerName="test-curl-11041-9c3805" containerID="cri-o://3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919" gracePeriod=2
Nov 03 20:11:02.182479 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:02.182380    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11041-3417e9" podStartSLOduration=1.182365749 podStartE2EDuration="1.182365749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:02.179933195 +0000 UTC m=+1844.692754979" watchObservedRunningTime="2026-11-03 20:11:02.182365749 +0000 UTC m=+1844.695187531"
Nov 03 20:11:02.324017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:02.323988    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-80ea48"
Nov 03 20:11:02.457609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:02.457575    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1360c5-2ba3-476b-8839-18e6e22c1683-kube-api-access-fcj8q\" (UniqueName: \"kubernetes.io/projected/bc1360c5-2ba3-476b-8839-18e6e22c1683-kube-api-access-fcj8q\") pod \"bc1360c5-2ba3-476b-8839-18e6e22c1683\" (UID: \"bc1360c5-2ba3-476b-8839-18e6e22c1683\") "
Nov 03 20:11:02.459833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:02.459807    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1360c5-2ba3-476b-8839-18e6e22c1683-kube-api-access-fcj8q" pod "bc1360c5-2ba3-476b-8839-18e6e22c1683" (UID: "bc1360c5-2ba3-476b-8839-18e6e22c1683"). InnerVolumeSpecName "kube-api-access-fcj8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:02.559884 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:02.559833    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcj8q\" (UniqueName: \"kubernetes.io/projected/bc1360c5-2ba3-476b-8839-18e6e22c1683-kube-api-access-fcj8q\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:02.857971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:02.857891    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11041-3417e9"]
Nov 03 20:11:02.864365 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:02.864328    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11041-3417e9"]
Nov 03 20:11:03.176491 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:03.176451    5600 generic.go:356] "Generic (PLEG): container finished" podID="bc1360c5-2ba3-476b-8839-18e6e22c1683" containerID="4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7" exitCode=137
Nov 03 20:11:03.176948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:03.176530    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-80ea48"
Nov 03 20:11:03.176948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:03.176577    5600 scope.go:122] "RemoveContainer" containerID="4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7"
Nov 03 20:11:03.188251 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:03.188226    5600 scope.go:122] "RemoveContainer" containerID="4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7"
Nov 03 20:11:03.188631 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:03.188608    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7\": container with ID starting with 4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7 not found: ID does not exist" containerID="4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7"
Nov 03 20:11:03.188716 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:03.188637    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7"} err="failed to get container status \"4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7\": rpc error: code = NotFound desc = could not find container \"4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7\": container with ID starting with 4de9c522c7833da0b3b90e8266f3b342803dbf4bc15121b8b55310f4bffef3a7 not found: ID does not exist"
Nov 03 20:11:03.188962 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:03.188946    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11041-80ea48\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="bc1360c5-2ba3-476b-8839-18e6e22c1683" pod="opendatahub/test-curl-11041-80ea48"
Nov 03 20:11:03.619808 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:03.619723    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1360c5-2ba3-476b-8839-18e6e22c1683" path="/var/lib/kubelet/pods/bc1360c5-2ba3-476b-8839-18e6e22c1683/volumes"
Nov 03 20:11:03.703517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:03.692381    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"]
Nov 03 20:11:03.703517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:03.692825    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6" podUID="2a0f3867-0e7f-4897-a8a7-652849d5799a" containerName="maas-api" containerID="cri-o://8fde46ca294c90691c4ffd440d41d7fc368b781b9c66075c186533e79407775a" gracePeriod=30
Nov 03 20:11:03.721409 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:03.721372    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6" podUID="2a0f3867-0e7f-4897-a8a7-652849d5799a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.89:8443/health\": dial tcp 10.134.0.89:8443: connect: connection refused"
Nov 03 20:11:04.514795 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:04.496591    5600 generic.go:356] "Generic (PLEG): container finished" podID="2a0f3867-0e7f-4897-a8a7-652849d5799a" containerID="8fde46ca294c90691c4ffd440d41d7fc368b781b9c66075c186533e79407775a" exitCode=0
Nov 03 20:11:04.514795 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:04.496948    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6" event={"ID":"2a0f3867-0e7f-4897-a8a7-652849d5799a","Type":"ContainerDied","Data":"8fde46ca294c90691c4ffd440d41d7fc368b781b9c66075c186533e79407775a"}
Nov 03 20:11:04.518234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:04.516352    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11041-3417e9" podUID="0e442543-de64-4941-a3c8-c13e2183fa27" containerName="test-curl-11041-3417e9" containerID="cri-o://e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c" gracePeriod=2
Nov 03 20:11:04.917459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:04.912546    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:11:05.127810 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.100030    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-maas-api-tls\") pod \"2a0f3867-0e7f-4897-a8a7-652849d5799a\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") "
Nov 03 20:11:05.127810 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.100228    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-metrics-certs\") pod \"2a0f3867-0e7f-4897-a8a7-652849d5799a\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") "
Nov 03 20:11:05.127810 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.100324    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0f3867-0e7f-4897-a8a7-652849d5799a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0f3867-0e7f-4897-a8a7-652849d5799a-openshift-service-ca\") pod \"2a0f3867-0e7f-4897-a8a7-652849d5799a\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") "
Nov 03 20:11:05.127810 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.100573    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0f3867-0e7f-4897-a8a7-652849d5799a-kube-api-access-445pf\" (UniqueName: \"kubernetes.io/projected/2a0f3867-0e7f-4897-a8a7-652849d5799a-kube-api-access-445pf\") pod \"2a0f3867-0e7f-4897-a8a7-652849d5799a\" (UID: \"2a0f3867-0e7f-4897-a8a7-652849d5799a\") "
Nov 03 20:11:05.149993 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.149135    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0f3867-0e7f-4897-a8a7-652849d5799a-openshift-service-ca" pod "2a0f3867-0e7f-4897-a8a7-652849d5799a" (UID: "2a0f3867-0e7f-4897-a8a7-652849d5799a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:05.150492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.150364    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-metrics-certs" pod "2a0f3867-0e7f-4897-a8a7-652849d5799a" (UID: "2a0f3867-0e7f-4897-a8a7-652849d5799a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:05.165480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.164918    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0f3867-0e7f-4897-a8a7-652849d5799a-kube-api-access-445pf" pod "2a0f3867-0e7f-4897-a8a7-652849d5799a" (UID: "2a0f3867-0e7f-4897-a8a7-652849d5799a"). InnerVolumeSpecName "kube-api-access-445pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:05.165480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.165210    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-maas-api-tls" pod "2a0f3867-0e7f-4897-a8a7-652849d5799a" (UID: "2a0f3867-0e7f-4897-a8a7-652849d5799a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:05.177465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.175992    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-9c3805"
Nov 03 20:11:05.207082 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.203770    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-445pf\" (UniqueName: \"kubernetes.io/projected/2a0f3867-0e7f-4897-a8a7-652849d5799a-kube-api-access-445pf\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:05.207082 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.203808    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:05.207082 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.203827    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a0f3867-0e7f-4897-a8a7-652849d5799a-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:05.207082 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.203841    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0f3867-0e7f-4897-a8a7-652849d5799a-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:05.305324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.305286    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd2ff8f-eb3d-4b89-9a36-c1860b271d55-kube-api-access-pjfxr\" (UniqueName: \"kubernetes.io/projected/ccd2ff8f-eb3d-4b89-9a36-c1860b271d55-kube-api-access-pjfxr\") pod \"ccd2ff8f-eb3d-4b89-9a36-c1860b271d55\" (UID: \"ccd2ff8f-eb3d-4b89-9a36-c1860b271d55\") "
Nov 03 20:11:05.307890 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.307851    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd2ff8f-eb3d-4b89-9a36-c1860b271d55-kube-api-access-pjfxr" pod "ccd2ff8f-eb3d-4b89-9a36-c1860b271d55" (UID: "ccd2ff8f-eb3d-4b89-9a36-c1860b271d55"). InnerVolumeSpecName "kube-api-access-pjfxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:05.407705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.407663    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjfxr\" (UniqueName: \"kubernetes.io/projected/ccd2ff8f-eb3d-4b89-9a36-c1860b271d55-kube-api-access-pjfxr\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:05.530911 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.530878    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"
Nov 03 20:11:05.531401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.530903    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6" event={"ID":"2a0f3867-0e7f-4897-a8a7-652849d5799a","Type":"ContainerDied","Data":"63ce59e5fea09b1d353504c986600aab74e11fa73454c7493c8284d9cff8fefc"}
Nov 03 20:11:05.531401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.530984    5600 scope.go:122] "RemoveContainer" containerID="8fde46ca294c90691c4ffd440d41d7fc368b781b9c66075c186533e79407775a"
Nov 03 20:11:05.534034 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.534003    5600 generic.go:356] "Generic (PLEG): container finished" podID="ccd2ff8f-eb3d-4b89-9a36-c1860b271d55" containerID="3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919" exitCode=137
Nov 03 20:11:05.534184 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.534059    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-9c3805"
Nov 03 20:11:05.552016 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.551994    5600 scope.go:122] "RemoveContainer" containerID="3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919"
Nov 03 20:11:05.565541 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.565371    5600 scope.go:122] "RemoveContainer" containerID="3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919"
Nov 03 20:11:05.566102 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:05.566068    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919\": container with ID starting with 3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919 not found: ID does not exist" containerID="3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919"
Nov 03 20:11:05.566212 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.566106    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919"} err="failed to get container status \"3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919\": rpc error: code = NotFound desc = could not find container \"3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919\": container with ID starting with 3f4d1fde2bf078a2d8a24fcc4d33d791c610027a44072c87ee696afe7b544919 not found: ID does not exist"
Nov 03 20:11:05.566607 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.566589    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"]
Nov 03 20:11:05.567412 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:05.567390    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11041-9c3805\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="ccd2ff8f-eb3d-4b89-9a36-c1860b271d55" pod="opendatahub/test-curl-11041-9c3805"
Nov 03 20:11:05.569822 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:05.569793    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11041-9c3805\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="ccd2ff8f-eb3d-4b89-9a36-c1860b271d55" pod="opendatahub/test-curl-11041-9c3805"
Nov 03 20:11:05.570857 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.570835    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-4pbl6"]
Nov 03 20:11:05.624354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.624315    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0f3867-0e7f-4897-a8a7-652849d5799a" path="/var/lib/kubelet/pods/2a0f3867-0e7f-4897-a8a7-652849d5799a/volumes"
Nov 03 20:11:05.626787 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:05.626208    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd2ff8f-eb3d-4b89-9a36-c1860b271d55" path="/var/lib/kubelet/pods/ccd2ff8f-eb3d-4b89-9a36-c1860b271d55/volumes"
Nov 03 20:11:06.848593 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:06.848559    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-3417e9"
Nov 03 20:11:06.954086 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:06.954048    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e442543-de64-4941-a3c8-c13e2183fa27-kube-api-access-p8fzv\" (UniqueName: \"kubernetes.io/projected/0e442543-de64-4941-a3c8-c13e2183fa27-kube-api-access-p8fzv\") pod \"0e442543-de64-4941-a3c8-c13e2183fa27\" (UID: \"0e442543-de64-4941-a3c8-c13e2183fa27\") "
Nov 03 20:11:06.956427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:06.956400    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e442543-de64-4941-a3c8-c13e2183fa27-kube-api-access-p8fzv" pod "0e442543-de64-4941-a3c8-c13e2183fa27" (UID: "0e442543-de64-4941-a3c8-c13e2183fa27"). InnerVolumeSpecName "kube-api-access-p8fzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:07.056702 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:07.056664    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8fzv\" (UniqueName: \"kubernetes.io/projected/0e442543-de64-4941-a3c8-c13e2183fa27-kube-api-access-p8fzv\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:07.556147 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:07.556102    5600 generic.go:356] "Generic (PLEG): container finished" podID="0e442543-de64-4941-a3c8-c13e2183fa27" containerID="e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c" exitCode=137
Nov 03 20:11:07.556410 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:07.556162    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11041-3417e9"
Nov 03 20:11:07.556410 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:07.556184    5600 scope.go:122] "RemoveContainer" containerID="e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c"
Nov 03 20:11:07.567419 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:07.567247    5600 scope.go:122] "RemoveContainer" containerID="e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c"
Nov 03 20:11:07.567625 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:07.567597    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11041-3417e9\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="0e442543-de64-4941-a3c8-c13e2183fa27" pod="opendatahub/test-curl-11041-3417e9"
Nov 03 20:11:07.567625 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:07.567609    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c\": container with ID starting with e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c not found: ID does not exist" containerID="e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c"
Nov 03 20:11:07.567709 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:07.567644    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c"} err="failed to get container status \"e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c\": rpc error: code = NotFound desc = could not find container \"e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c\": container with ID starting with e38c7277a35fb43aebafe037809edde5573813f1cacf0f5cdaeb5454bcd1ff4c not found: ID does not exist"
Nov 03 20:11:07.618119 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:07.618080    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11041-3417e9\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="0e442543-de64-4941-a3c8-c13e2183fa27" pod="opendatahub/test-curl-11041-3417e9"
Nov 03 20:11:07.619928 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:07.619905    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e442543-de64-4941-a3c8-c13e2183fa27" path="/var/lib/kubelet/pods/0e442543-de64-4941-a3c8-c13e2183fa27/volumes"
Nov 03 20:11:13.983461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:13.982572    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"]
Nov 03 20:11:13.983992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:13.983665    5600 cpu_manager.go:414] "containerMap: removing container" podUID="2a0f3867-0e7f-4897-a8a7-652849d5799a" containerName="maas-api"
Nov 03 20:11:13.983992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:13.983686    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0f3867-0e7f-4897-a8a7-652849d5799a" containerName="maas-api"
Nov 03 20:11:13.983992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:13.983728    5600 cpu_manager.go:414] "containerMap: removing container" podUID="0e442543-de64-4941-a3c8-c13e2183fa27" containerName="test-curl-11041-3417e9"
Nov 03 20:11:13.983992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:13.983737    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e442543-de64-4941-a3c8-c13e2183fa27" containerName="test-curl-11041-3417e9"
Nov 03 20:11:13.983992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:13.983928    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0f3867-0e7f-4897-a8a7-652849d5799a" containerName="maas-api"
Nov 03 20:11:13.983992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:13.983946    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e442543-de64-4941-a3c8-c13e2183fa27" containerName="test-curl-11041-3417e9"
Nov 03 20:11:14.090460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.089811    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.116461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.113374    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"]
Nov 03 20:11:14.129462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.119025    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-433102-serving-cert\""
Nov 03 20:11:14.234755 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.180459    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-metrics-certs\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.234755 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.180546    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsj46\" (UniqueName: \"kubernetes.io/projected/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-kube-api-access-qsj46\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.234755 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.180589    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.234755 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.180791    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-maas-api-tls\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.283477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.283401    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-metrics-certs\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.283792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.283767    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsj46\" (UniqueName: \"kubernetes.io/projected/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-kube-api-access-qsj46\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.283938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.283923    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.284215 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.284198    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-maas-api-tls\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.284616 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:14.284595    5600 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-433102-serving-cert: secret "maas-api-e2e-api-a-433102-serving-cert" not found
Nov 03 20:11:14.284817 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:14.284805    5600 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-maas-api-tls podName:ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c nodeName:}" failed. No retries permitted until 2026-11-03 20:11:14.784769607 +0000 UTC m=+1857.297591391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-maas-api-tls") pod "maas-api-e2e-api-a-433102-67cd5c5694-ns8dr" (UID: "ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c") : secret "maas-api-e2e-api-a-433102-serving-cert" not found
Nov 03 20:11:14.289493 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.288312    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-metrics-certs\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.289493 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.289245    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.331980 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.331946    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsj46\" (UniqueName: \"kubernetes.io/projected/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-kube-api-access-qsj46\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.793957 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.793916    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-maas-api-tls\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.799077 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.799045    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-maas-api-tls\") pod \"maas-api-e2e-api-a-433102-67cd5c5694-ns8dr\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:14.829106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:14.829061    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:15.365826 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:15.365730    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"]
Nov 03 20:11:15.373406 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:11:15.373347    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe46c2c_73de_49c5_b8c4_cb69c7b8cc1c.slice/crio-8905113c7591af229f3ed9200573645f692abfa0f9c8b28d8d24723e5ba18083 WatchSource:0}: Error finding container 8905113c7591af229f3ed9200573645f692abfa0f9c8b28d8d24723e5ba18083: Status 404 returned error can't find the container with id 8905113c7591af229f3ed9200573645f692abfa0f9c8b28d8d24723e5ba18083
Nov 03 20:11:15.703460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:15.702653    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr" event={"ID":"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c","Type":"ContainerStarted","Data":"8905113c7591af229f3ed9200573645f692abfa0f9c8b28d8d24723e5ba18083"}
Nov 03 20:11:16.767459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:16.756760    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr" event={"ID":"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c","Type":"ContainerStarted","Data":"1a35ef16a6d5f9895f43f2fb61c319476cb133f84eab26bd49acf89f81d308fc"}
Nov 03 20:11:16.767459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:16.758197    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:16.795470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:16.793020    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr" podStartSLOduration=3.246962345 podStartE2EDuration="3.7930005s" totalImagesPullingTime="546.038155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:16.784126283 +0000 UTC m=+1859.296948065" watchObservedRunningTime="2026-11-03 20:11:16.7930005 +0000 UTC m=+1859.305822283"
Nov 03 20:11:19.082461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:19.078675    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j" podUID="8316a979-44c7-4b82-8df4-1a1ced93d9a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:22.970067 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:22.970022    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"]
Nov 03 20:11:22.970529 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:22.970323    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy" containerID="cri-o://5ac7cd21b046bb4e8b5623247df257f3919ec7c08b52e7c704f984ed67b2d079" gracePeriod=30
Nov 03 20:11:22.976271 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:22.976246    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"]
Nov 03 20:11:22.976500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:22.976454    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy" containerID="cri-o://7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e" gracePeriod=30
Nov 03 20:11:24.466771 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:24.466739    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:11:25.138939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.138902    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"]
Nov 03 20:11:25.139167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.139140    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd" podUID="31343dd5-bf61-42bc-9b78-3fcf989cad5b" containerName="maas-api" containerID="cri-o://5933ee95887b4fed2cade3445f6adfe3cff167436ddf27f10fb31205784264b3" gracePeriod=30
Nov 03 20:11:25.785993 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.784982    5600 generic.go:356] "Generic (PLEG): container finished" podID="31343dd5-bf61-42bc-9b78-3fcf989cad5b" containerID="5933ee95887b4fed2cade3445f6adfe3cff167436ddf27f10fb31205784264b3" exitCode=0
Nov 03 20:11:25.785993 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.785273    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd" event={"ID":"31343dd5-bf61-42bc-9b78-3fcf989cad5b","Type":"ContainerDied","Data":"5933ee95887b4fed2cade3445f6adfe3cff167436ddf27f10fb31205784264b3"}
Nov 03 20:11:25.842676 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.842646    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:11:25.878924 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.878892    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"]
Nov 03 20:11:25.879976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.879948    5600 cpu_manager.go:414] "containerMap: removing container" podUID="31343dd5-bf61-42bc-9b78-3fcf989cad5b" containerName="maas-api"
Nov 03 20:11:25.879976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.879975    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31343dd5-bf61-42bc-9b78-3fcf989cad5b" containerName="maas-api"
Nov 03 20:11:25.880293 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.880217    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="31343dd5-bf61-42bc-9b78-3fcf989cad5b" containerName="maas-api"
Nov 03 20:11:25.911458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.884223    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31343dd5-bf61-42bc-9b78-3fcf989cad5b-kube-api-access-7g8hb\" (UniqueName: \"kubernetes.io/projected/31343dd5-bf61-42bc-9b78-3fcf989cad5b-kube-api-access-7g8hb\") pod \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") "
Nov 03 20:11:25.911458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.884266    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31343dd5-bf61-42bc-9b78-3fcf989cad5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31343dd5-bf61-42bc-9b78-3fcf989cad5b-openshift-service-ca\") pod \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") "
Nov 03 20:11:25.911458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.884407    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-metrics-certs\") pod \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") "
Nov 03 20:11:25.911458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.884543    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-maas-api-tls\") pod \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\" (UID: \"31343dd5-bf61-42bc-9b78-3fcf989cad5b\") "
Nov 03 20:11:25.911458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.887139    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31343dd5-bf61-42bc-9b78-3fcf989cad5b-openshift-service-ca" pod "31343dd5-bf61-42bc-9b78-3fcf989cad5b" (UID: "31343dd5-bf61-42bc-9b78-3fcf989cad5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:25.911458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.890474    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:25.911458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.904990    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"]
Nov 03 20:11:25.936460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.929655    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-metrics-certs" pod "31343dd5-bf61-42bc-9b78-3fcf989cad5b" (UID: "31343dd5-bf61-42bc-9b78-3fcf989cad5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:25.936460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.929713    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31343dd5-bf61-42bc-9b78-3fcf989cad5b-kube-api-access-7g8hb" pod "31343dd5-bf61-42bc-9b78-3fcf989cad5b" (UID: "31343dd5-bf61-42bc-9b78-3fcf989cad5b"). InnerVolumeSpecName "kube-api-access-7g8hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:25.936797 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.936750    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-maas-api-tls" pod "31343dd5-bf61-42bc-9b78-3fcf989cad5b" (UID: "31343dd5-bf61-42bc-9b78-3fcf989cad5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:25.941644 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.941284    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-cad531-openshift-default-dockercfg-gj4fh\""
Nov 03 20:11:25.986930 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.986878    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:25.987175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.986988    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:25.987175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.987028    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:25.987175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.987074    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:25.987369 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.987227    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvk7g\" (UniqueName: \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-kube-api-access-qvk7g\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:25.987369 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.987357    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:25.987516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.987427    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:25.987575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.987513    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:25.987575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.987561    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:25.987692 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.987663    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g8hb\" (UniqueName: \"kubernetes.io/projected/31343dd5-bf61-42bc-9b78-3fcf989cad5b-kube-api-access-7g8hb\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:25.987794 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.987769    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31343dd5-bf61-42bc-9b78-3fcf989cad5b-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:25.987860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.987802    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:25.987860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:25.987818    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31343dd5-bf61-42bc-9b78-3fcf989cad5b-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:26.089720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.089623    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.089917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.089726    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.089917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.089759    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.089917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.089836    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.090120 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.090024    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.090120 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.090086    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.090235 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.090123    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.090235 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.090169    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.090335 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.090274    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvk7g\" (UniqueName: \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-kube-api-access-qvk7g\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.091127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.091098    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.091860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.091833    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.092064 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.092037    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.092361 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.092325    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.092570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.092508    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.095200 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.095173    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.095292 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.095273    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.113034 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.113001    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvk7g\" (UniqueName: \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-kube-api-access-qvk7g\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.114176 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.114141    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.280189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.280152    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:26.589379 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.589347    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"]
Nov 03 20:11:26.591771 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:11:26.591732    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c3f78a_9dc4_463d_aa67_3d634d6caa75.slice/crio-e9edfe949cb511e1b9a72e44785e8da3ad8568998e2ccabc5010be7020dcaf1f WatchSource:0}: Error finding container e9edfe949cb511e1b9a72e44785e8da3ad8568998e2ccabc5010be7020dcaf1f: Status 404 returned error can't find the container with id e9edfe949cb511e1b9a72e44785e8da3ad8568998e2ccabc5010be7020dcaf1f
Nov 03 20:11:26.595579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.595541    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:11:26.595699 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.595614    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:11:26.595699 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.595644    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:11:26.803905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.801708    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd" event={"ID":"31343dd5-bf61-42bc-9b78-3fcf989cad5b","Type":"ContainerDied","Data":"dc6c87b5b0a2ed7b114eeee655eba561c2832b7b74a258cd9e9d561b9b7bf1c1"}
Nov 03 20:11:26.803905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.801784    5600 scope.go:122] "RemoveContainer" containerID="5933ee95887b4fed2cade3445f6adfe3cff167436ddf27f10fb31205784264b3"
Nov 03 20:11:26.803905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.801969    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"
Nov 03 20:11:26.821000 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.820950    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" event={"ID":"17c3f78a-9dc4-463d-aa67-3d634d6caa75","Type":"ContainerStarted","Data":"4c9ace9af392833b150f893b682f72ede76f280d961d00f3c1a868db89ab16b2"}
Nov 03 20:11:26.821211 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.821190    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" event={"ID":"17c3f78a-9dc4-463d-aa67-3d634d6caa75","Type":"ContainerStarted","Data":"e9edfe949cb511e1b9a72e44785e8da3ad8568998e2ccabc5010be7020dcaf1f"}
Nov 03 20:11:26.840025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.840004    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"]
Nov 03 20:11:26.845384 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.845332    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1962f9-6fbdc8446-wpzxd"]
Nov 03 20:11:26.863862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:26.863807    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" podStartSLOduration=8.863787186 podStartE2EDuration="8.863787186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:26.861995147 +0000 UTC m=+1869.374816930" watchObservedRunningTime="2026-11-03 20:11:26.863787186 +0000 UTC m=+1869.376608966"
Nov 03 20:11:27.280902 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:27.280871    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:27.282400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:27.282374    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:11:27.282541 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:27.282445    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 20:11:27.640446 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:27.640404    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31343dd5-bf61-42bc-9b78-3fcf989cad5b" path="/var/lib/kubelet/pods/31343dd5-bf61-42bc-9b78-3fcf989cad5b/volumes"
Nov 03 20:11:28.281183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.281097    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:11:28.293500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.289555    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 20:11:28.466910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.466880    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:11:28.547788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.545631    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"]
Nov 03 20:11:28.547788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.546905    5600 cpu_manager.go:414] "containerMap: removing container" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy"
Nov 03 20:11:28.547788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.546926    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy"
Nov 03 20:11:28.547788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.547121    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerName="istio-proxy"
Nov 03 20:11:28.552129 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.552072    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.553261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.553237    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"]
Nov 03 20:11:28.556183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.555664    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-certs\") pod \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") "
Nov 03 20:11:28.556183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.555731    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-data\") pod \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") "
Nov 03 20:11:28.556183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.555771    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-credential-socket\") pod \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") "
Nov 03 20:11:28.556183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.555803    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-kube-api-access-8nk5t\" (UniqueName: \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-kube-api-access-8nk5t\") pod \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") "
Nov 03 20:11:28.556183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.555879    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-token\" (UniqueName: \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-token\") pod \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") "
Nov 03 20:11:28.556183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.555938    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-socket\") pod \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") "
Nov 03 20:11:28.556183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.555959    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-certs" pod "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" (UID: "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:28.556183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.555966    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-envoy\") pod \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") "
Nov 03 20:11:28.556183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556078    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-podinfo\") pod \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") "
Nov 03 20:11:28.556183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556134    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istiod-ca-cert\") pod \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\" (UID: \"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a\") "
Nov 03 20:11:28.556654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556310    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.556654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556364    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.556654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556403    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65c57ce-956a-4eba-a096-19d49a17b0aa-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.556654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556474    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.556654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556523    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.556654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556549    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.556654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556605    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsc74\" (UniqueName: \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-kube-api-access-fsc74\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.556879 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556698    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.556879 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556727    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.556879 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.556860    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.559242 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.559060    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-data" pod "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" (UID: "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:28.559242 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.559200    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istiod-ca-cert" pod "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" (UID: "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:28.559908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.559510    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-envoy" pod "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" (UID: "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:28.564728 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.564603    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-socket" pod "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" (UID: "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:28.564864 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.564760    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-podinfo" pod "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" (UID: "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:11:28.565862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.565837    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-credential-socket" pod "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" (UID: "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:28.566777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.566748    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-kube-api-access-8nk5t" pod "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" (UID: "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a"). InnerVolumeSpecName "kube-api-access-8nk5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:28.571951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.569035    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-token" pod "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" (UID: "bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:28.658425 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.658379    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsc74\" (UniqueName: \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-kube-api-access-fsc74\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.658651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.658537    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.658651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.658595    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.658774 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.658758    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.658862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.658838    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.659316 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659285    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.659468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659150    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.659468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659334    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65c57ce-956a-4eba-a096-19d49a17b0aa-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.659468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659389    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.659468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659460    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.659707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659489    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.659707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659569    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.659707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659580    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.659707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659590    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.659707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659599    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.659707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659607    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.659707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659616    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.659707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659625    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.659707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659623    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.659707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659633    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nk5t\" (UniqueName: \"kubernetes.io/projected/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a-kube-api-access-8nk5t\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.660204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659894    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.660204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.659979    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65c57ce-956a-4eba-a096-19d49a17b0aa-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.662113 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.662092    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.662214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.662138    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.673272 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.673214    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.674265 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.673925    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsc74\" (UniqueName: \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-kube-api-access-fsc74\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.871506 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.871476    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:11:28.872634 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.872607    5600 generic.go:356] "Generic (PLEG): container finished" podID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" containerID="7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e" exitCode=0
Nov 03 20:11:28.872773 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.872670    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"
Nov 03 20:11:28.872895 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.872876    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" event={"ID":"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a","Type":"ContainerDied","Data":"7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e"}
Nov 03 20:11:28.872956 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.872932    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9" event={"ID":"bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a","Type":"ContainerDied","Data":"36f11c91944d219c87b278176cc07ddfaf99d72c2099eee504ca8f962245b7ca"}
Nov 03 20:11:28.873011 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.872966    5600 scope.go:122] "RemoveContainer" containerID="7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e"
Nov 03 20:11:28.876895 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.876866    5600 generic.go:356] "Generic (PLEG): container finished" podID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerID="5ac7cd21b046bb4e8b5623247df257f3919ec7c08b52e7c704f984ed67b2d079" exitCode=0
Nov 03 20:11:28.877397 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.876901    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m" event={"ID":"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf","Type":"ContainerDied","Data":"5ac7cd21b046bb4e8b5623247df257f3919ec7c08b52e7c704f984ed67b2d079"}
Nov 03 20:11:28.879256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.876911    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"
Nov 03 20:11:28.934747 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.934249    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:28.940933 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:28.938839    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:11:28.958277 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.958248    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"]
Nov 03 20:11:28.960932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.960908    5600 cpu_manager.go:414] "containerMap: removing container" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy"
Nov 03 20:11:28.961106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.961093    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy"
Nov 03 20:11:28.961462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.961425    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" containerName="istio-proxy"
Nov 03 20:11:28.965612 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.965562    5600 scope.go:122] "RemoveContainer" containerID="7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e"
Nov 03 20:11:28.966250 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:28.966085    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e\": container with ID starting with 7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e not found: ID does not exist" containerID="7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e"
Nov 03 20:11:28.966250 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.966123    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e"} err="failed to get container status \"7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e\": rpc error: code = NotFound desc = could not find container \"7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e\": container with ID starting with 7ea5a60a9dac0cf4cc539a2c8f9a26b60c57ec6a53d0a83102c784bfda8dac9e not found: ID does not exist"
Nov 03 20:11:28.966250 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.966144    5600 scope.go:122] "RemoveContainer" containerID="5ac7cd21b046bb4e8b5623247df257f3919ec7c08b52e7c704f984ed67b2d079"
Nov 03 20:11:28.968760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.967667    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-certs\") pod \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") "
Nov 03 20:11:28.968760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.967731    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-socket\") pod \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") "
Nov 03 20:11:28.968760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.967763    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istiod-ca-cert\") pod \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") "
Nov 03 20:11:28.968760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.967861    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-kube-api-access-gbsnq\" (UniqueName: \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-kube-api-access-gbsnq\") pod \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") "
Nov 03 20:11:28.968760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.967973    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-credential-socket\") pod \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") "
Nov 03 20:11:28.968760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.968077    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-token\" (UniqueName: \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-token\") pod \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") "
Nov 03 20:11:28.968760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.968114    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-envoy\") pod \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") "
Nov 03 20:11:28.968760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.968152    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-data\") pod \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") "
Nov 03 20:11:28.968760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.968244    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-socket" pod "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" (UID: "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:28.968760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.968506    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-certs" pod "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" (UID: "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:28.968760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.968648    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-credential-socket" pod "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" (UID: "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:28.969305 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.968837    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istiod-ca-cert" pod "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" (UID: "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:28.970472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.969586    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-podinfo\") pod \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\" (UID: \"8fb8ac5a-ecb6-4e61-91d3-5877bad385cf\") "
Nov 03 20:11:28.971402 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.971020    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-data" pod "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" (UID: "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:28.971402 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.971158    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-kube-api-access-gbsnq" pod "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" (UID: "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf"). InnerVolumeSpecName "kube-api-access-gbsnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:28.971402 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.971361    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.971402 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.971382    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.971402 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.971398    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.971725 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.971413    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:28.974224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.974168    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-podinfo" pod "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" (UID: "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:11:28.974224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.974184    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-envoy" pod "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" (UID: "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:28.975804 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.975785    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"]
Nov 03 20:11:28.975919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.975828    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"]
Nov 03 20:11:28.975981 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.975918    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.015675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.980237    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-acd98d21-serving-cert\""
Nov 03 20:11:29.015675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.981010    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-token" pod "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" (UID: "8fb8ac5a-ecb6-4e61-91d3-5877bad385cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:29.015675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:28.981707    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-x8gx9"]
Nov 03 20:11:29.074216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.072995    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-metrics-certs\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.074216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.073219    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dp9h\" (UniqueName: \"kubernetes.io/projected/94221571-ec07-465a-b972-02f0a50dc05b-kube-api-access-4dp9h\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.074216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.073407    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94221571-ec07-465a-b972-02f0a50dc05b-openshift-service-ca\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.074216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.073546    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-maas-api-tls\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.074216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.073675    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:29.074216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.073692    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:29.074216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.073707    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:29.074216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.073720    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:29.074216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.073733    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbsnq\" (UniqueName: \"kubernetes.io/projected/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf-kube-api-access-gbsnq\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:29.176462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.176414    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94221571-ec07-465a-b972-02f0a50dc05b-openshift-service-ca\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.176649 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.176520    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-maas-api-tls\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.176649 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.176636    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-metrics-certs\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.176785 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.176768    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dp9h\" (UniqueName: \"kubernetes.io/projected/94221571-ec07-465a-b972-02f0a50dc05b-kube-api-access-4dp9h\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.177291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.177262    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94221571-ec07-465a-b972-02f0a50dc05b-openshift-service-ca\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.180165 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.180139    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-maas-api-tls\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.180288 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.180242    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-metrics-certs\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.222191 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.222143    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dp9h\" (UniqueName: \"kubernetes.io/projected/94221571-ec07-465a-b972-02f0a50dc05b-kube-api-access-4dp9h\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.256505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.256473    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"]
Nov 03 20:11:29.260877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.260850    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-z9t6m"]
Nov 03 20:11:29.281230 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.281202    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:11:29.281656 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.281262    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 20:11:29.360998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.360968    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:29.362237 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:11:29.362209    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65c57ce_956a_4eba_a096_19d49a17b0aa.slice/crio-c086f6bd16c718e6f6f35219f773350393c88b12cba965d0c57fd7f11a49b370 WatchSource:0}: Error finding container c086f6bd16c718e6f6f35219f773350393c88b12cba965d0c57fd7f11a49b370: Status 404 returned error can't find the container with id c086f6bd16c718e6f6f35219f773350393c88b12cba965d0c57fd7f11a49b370
Nov 03 20:11:29.362477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.362451    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"]
Nov 03 20:11:29.365401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.365373    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:11:29.365497 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.365456    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:11:29.365497 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.365486    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:11:29.773366 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.773288    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb8ac5a-ecb6-4e61-91d3-5877bad385cf" path="/var/lib/kubelet/pods/8fb8ac5a-ecb6-4e61-91d3-5877bad385cf/volumes"
Nov 03 20:11:29.775955 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.775915    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a" path="/var/lib/kubelet/pods/bd9a468e-01e6-45f1-b0df-4f1fe4c9f54a/volumes"
Nov 03 20:11:29.776591 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.776569    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"]
Nov 03 20:11:29.823341 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:11:29.823301    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94221571_ec07_465a_b972_02f0a50dc05b.slice/crio-db18c1e6448a4286de1bc4e5def1a18036b741e65ad9c596f9d8d263ffdfc8ee WatchSource:0}: Error finding container db18c1e6448a4286de1bc4e5def1a18036b741e65ad9c596f9d8d263ffdfc8ee: Status 404 returned error can't find the container with id db18c1e6448a4286de1bc4e5def1a18036b741e65ad9c596f9d8d263ffdfc8ee
Nov 03 20:11:29.942407 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.942367    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" event={"ID":"e65c57ce-956a-4eba-a096-19d49a17b0aa","Type":"ContainerStarted","Data":"0cf72cacee8fb0555bcda14c57f315ff1620bf8e12829458165d7a7a2431f1ae"}
Nov 03 20:11:29.944500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.944399    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" event={"ID":"e65c57ce-956a-4eba-a096-19d49a17b0aa","Type":"ContainerStarted","Data":"c086f6bd16c718e6f6f35219f773350393c88b12cba965d0c57fd7f11a49b370"}
Nov 03 20:11:29.947048 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.947023    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch" event={"ID":"94221571-ec07-465a-b972-02f0a50dc05b","Type":"ContainerStarted","Data":"db18c1e6448a4286de1bc4e5def1a18036b741e65ad9c596f9d8d263ffdfc8ee"}
Nov 03 20:11:29.966821 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:29.966755    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" podStartSLOduration=11.966735429 podStartE2EDuration="11.966735429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:29.96421131 +0000 UTC m=+1872.477033091" watchObservedRunningTime="2026-11-03 20:11:29.966735429 +0000 UTC m=+1872.479557222"
Nov 03 20:11:30.303463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:30.298304    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:11:30.303463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:30.298362    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 20:11:30.942468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:30.940981    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:30.942678 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:30.942658    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:11:30.942748 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:30.942715    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" podUID="e65c57ce-956a-4eba-a096-19d49a17b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:11:31.291458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:31.281866    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:11:31.291458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:31.281925    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 20:11:31.941570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:31.941534    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:11:31.942137 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:31.942106    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" podUID="e65c57ce-956a-4eba-a096-19d49a17b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:11:32.221825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:32.221621    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch" event={"ID":"94221571-ec07-465a-b972-02f0a50dc05b","Type":"ContainerStarted","Data":"f57e1abb4b71bb441898071049d1688b1917802d338620a18d28846a97f6de9e"}
Nov 03 20:11:32.221825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:32.221718    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:32.268827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:32.268710    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch" podStartSLOduration=9.537760448 podStartE2EDuration="10.268688983s" totalImagesPullingTime="730.928535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:32.263196638 +0000 UTC m=+1874.776018433" watchObservedRunningTime="2026-11-03 20:11:32.268688983 +0000 UTC m=+1874.781510767"
Nov 03 20:11:32.280864 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:32.280831    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:11:32.281032 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:32.280890    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 20:11:32.992459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:32.991914    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:11:32.992459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:32.991981    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" podUID="e65c57ce-956a-4eba-a096-19d49a17b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:11:33.280802 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:33.280716    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 20:11:33.281047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:33.281016    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 20:11:33.940570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:33.940530    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:11:33.940783 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:33.940595    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" podUID="e65c57ce-956a-4eba-a096-19d49a17b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:11:34.294855 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:34.294773    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:34.297076 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:34.297053    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:34.298398 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:34.298379    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:11:34.938497 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:34.938405    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:11:34.938875 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:34.938791    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" podUID="e65c57ce-956a-4eba-a096-19d49a17b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:11:35.938356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:35.938324    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:11:35.938788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:35.938381    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" podUID="e65c57ce-956a-4eba-a096-19d49a17b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:11:37.036598 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:37.036541    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:37.037131 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:37.036963    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:37.037923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:37.037903    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:11:38.406047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:38.406008    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:11:43.245704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:43.234605    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"]
Nov 03 20:11:43.245704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:43.234844    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws" podUID="fae5f00e-8af6-410b-b148-a08cb888051b" containerName="maas-api" containerID="cri-o://29e22e5844236cc8aba5ba78121dd4a682deab4017c16d1652931f667a087288" gracePeriod=30
Nov 03 20:11:43.867458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:43.865751    5600 generic.go:356] "Generic (PLEG): container finished" podID="fae5f00e-8af6-410b-b148-a08cb888051b" containerID="29e22e5844236cc8aba5ba78121dd4a682deab4017c16d1652931f667a087288" exitCode=0
Nov 03 20:11:43.867458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:43.865926    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws" event={"ID":"fae5f00e-8af6-410b-b148-a08cb888051b","Type":"ContainerDied","Data":"29e22e5844236cc8aba5ba78121dd4a682deab4017c16d1652931f667a087288"}
Nov 03 20:11:43.915457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:43.913824    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:11:44.049262 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.049221    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae5f00e-8af6-410b-b148-a08cb888051b-kube-api-access-bbmvp\" (UniqueName: \"kubernetes.io/projected/fae5f00e-8af6-410b-b148-a08cb888051b-kube-api-access-bbmvp\") pod \"fae5f00e-8af6-410b-b148-a08cb888051b\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") "
Nov 03 20:11:44.049555 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.049341    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-metrics-certs\") pod \"fae5f00e-8af6-410b-b148-a08cb888051b\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") "
Nov 03 20:11:44.049555 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.049525    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae5f00e-8af6-410b-b148-a08cb888051b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae5f00e-8af6-410b-b148-a08cb888051b-openshift-service-ca\") pod \"fae5f00e-8af6-410b-b148-a08cb888051b\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") "
Nov 03 20:11:44.049897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.049618    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-maas-api-tls\") pod \"fae5f00e-8af6-410b-b148-a08cb888051b\" (UID: \"fae5f00e-8af6-410b-b148-a08cb888051b\") "
Nov 03 20:11:44.050018 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.049994    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae5f00e-8af6-410b-b148-a08cb888051b-openshift-service-ca" pod "fae5f00e-8af6-410b-b148-a08cb888051b" (UID: "fae5f00e-8af6-410b-b148-a08cb888051b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:44.050528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.050460    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae5f00e-8af6-410b-b148-a08cb888051b-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:44.057340 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.057241    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-metrics-certs" pod "fae5f00e-8af6-410b-b148-a08cb888051b" (UID: "fae5f00e-8af6-410b-b148-a08cb888051b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:44.057340 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.057290    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-maas-api-tls" pod "fae5f00e-8af6-410b-b148-a08cb888051b" (UID: "fae5f00e-8af6-410b-b148-a08cb888051b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:44.057996 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.057970    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae5f00e-8af6-410b-b148-a08cb888051b-kube-api-access-bbmvp" pod "fae5f00e-8af6-410b-b148-a08cb888051b" (UID: "fae5f00e-8af6-410b-b148-a08cb888051b"). InnerVolumeSpecName "kube-api-access-bbmvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:44.151677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.151644    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:44.151677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.151673    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fae5f00e-8af6-410b-b148-a08cb888051b-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:44.151677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.151685    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbmvp\" (UniqueName: \"kubernetes.io/projected/fae5f00e-8af6-410b-b148-a08cb888051b-kube-api-access-bbmvp\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:44.881813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.881780    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws" event={"ID":"fae5f00e-8af6-410b-b148-a08cb888051b","Type":"ContainerDied","Data":"0d803cb2029ff442a2d0098e8686fe6e0e99af44b4e783eebc3a1408c09f505c"}
Nov 03 20:11:44.882234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.881839    5600 scope.go:122] "RemoveContainer" containerID="29e22e5844236cc8aba5ba78121dd4a682deab4017c16d1652931f667a087288"
Nov 03 20:11:44.882234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.881969    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"
Nov 03 20:11:44.921453 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.921373    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"]
Nov 03 20:11:44.930724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:44.929200    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9dc48-6cc55d86f4-hlmws"]
Nov 03 20:11:45.620825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:45.620789    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae5f00e-8af6-410b-b148-a08cb888051b" path="/var/lib/kubelet/pods/fae5f00e-8af6-410b-b148-a08cb888051b/volumes"
Nov 03 20:11:49.842067 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:49.842024    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11044-b8ed5a"]
Nov 03 20:11:49.844213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:49.843724    5600 cpu_manager.go:414] "containerMap: removing container" podUID="fae5f00e-8af6-410b-b148-a08cb888051b" containerName="maas-api"
Nov 03 20:11:49.844423 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:49.844404    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae5f00e-8af6-410b-b148-a08cb888051b" containerName="maas-api"
Nov 03 20:11:49.844836 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:49.844818    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae5f00e-8af6-410b-b148-a08cb888051b" containerName="maas-api"
Nov 03 20:11:49.852050 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:49.850344    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11044-b8ed5a"
Nov 03 20:11:49.861484 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:49.860537    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11044-b8ed5a"]
Nov 03 20:11:49.889420 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:49.889391    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4mfkw\""
Nov 03 20:11:49.927372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:49.927289    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ztp\" (UniqueName: \"kubernetes.io/projected/b231f0ef-7f0f-4142-a407-b035bc407f68-kube-api-access-w4ztp\") pod \"test-curl-11044-b8ed5a\" (UID: \"b231f0ef-7f0f-4142-a407-b035bc407f68\") " pod="opendatahub/test-curl-11044-b8ed5a"
Nov 03 20:11:50.029553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:50.029513    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ztp\" (UniqueName: \"kubernetes.io/projected/b231f0ef-7f0f-4142-a407-b035bc407f68-kube-api-access-w4ztp\") pod \"test-curl-11044-b8ed5a\" (UID: \"b231f0ef-7f0f-4142-a407-b035bc407f68\") " pod="opendatahub/test-curl-11044-b8ed5a"
Nov 03 20:11:50.043376 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:50.043337    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ztp\" (UniqueName: \"kubernetes.io/projected/b231f0ef-7f0f-4142-a407-b035bc407f68-kube-api-access-w4ztp\") pod \"test-curl-11044-b8ed5a\" (UID: \"b231f0ef-7f0f-4142-a407-b035bc407f68\") " pod="opendatahub/test-curl-11044-b8ed5a"
Nov 03 20:11:50.234160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:50.208380    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11044-b8ed5a"
Nov 03 20:11:51.017478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:50.998318    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11044-b8ed5a"]
Nov 03 20:11:51.095735 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:51.095653    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11044-b8ed5a" event={"ID":"b231f0ef-7f0f-4142-a407-b035bc407f68","Type":"ContainerStarted","Data":"cc4fa7cff5a1fae03915dea516c760e5793f654094e70edb950043328b5cf881"}
Nov 03 20:11:52.076330 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.076293    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"]
Nov 03 20:11:52.087667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.087639    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.101531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.101498    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"]
Nov 03 20:11:52.104167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.104139    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-43b780-openshift-default-dockercfg-95lzb\""
Nov 03 20:11:52.129462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.122231    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11044-b8ed5a" event={"ID":"b231f0ef-7f0f-4142-a407-b035bc407f68","Type":"ContainerStarted","Data":"c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65"}
Nov 03 20:11:52.133533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.130638    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-envoy\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.133533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.130801    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-certs\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.133533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.130851    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-data\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.133533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.131032    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.133533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.131077    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-token\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.133533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.131107    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-podinfo\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.133533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.131168    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788b5e72-bede-4a35-9a85-a1b2a0174c30-istiod-ca-cert\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.133533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.131209    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f56d\" (UniqueName: \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-kube-api-access-2f56d\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.133533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.131261    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-credential-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.152827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.152760    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11044-b8ed5a" podStartSLOduration=3.152740212 podStartE2EDuration="3.152740212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:52.145732952 +0000 UTC m=+1894.658554734" watchObservedRunningTime="2026-11-03 20:11:52.152740212 +0000 UTC m=+1894.665561998"
Nov 03 20:11:52.235073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.233761    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788b5e72-bede-4a35-9a85-a1b2a0174c30-istiod-ca-cert\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.235073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.234959    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f56d\" (UniqueName: \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-kube-api-access-2f56d\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.235073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.235040    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-credential-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.235399 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.235202    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-envoy\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.235399 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.235339    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-certs\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.235528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.235398    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-data\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.241876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.235649    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.241876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.235729    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-token\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.241876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.236333    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-podinfo\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.241876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.237228    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788b5e72-bede-4a35-9a85-a1b2a0174c30-istiod-ca-cert\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.241876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.237765    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-data\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.241876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.238026    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-credential-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.241876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.238302    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.241876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.238760    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-certs\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.280539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.280421    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-envoy\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.292027 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.290963    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-token\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.292027 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.291567    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f56d\" (UniqueName: \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-kube-api-access-2f56d\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.292027 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.291972    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-podinfo\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.440479 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.439754    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:52.927240 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.927191    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"]
Nov 03 20:11:52.936412 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:11:52.936381    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788b5e72_bede_4a35_9a85_a1b2a0174c30.slice/crio-af214b04d5b8bcae2a53cc2491a52d0a58cfa7dedd68ff2d7a515db917d56712 WatchSource:0}: Error finding container af214b04d5b8bcae2a53cc2491a52d0a58cfa7dedd68ff2d7a515db917d56712: Status 404 returned error can't find the container with id af214b04d5b8bcae2a53cc2491a52d0a58cfa7dedd68ff2d7a515db917d56712
Nov 03 20:11:52.939399 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.939351    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:11:52.939641 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.939617    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:11:52.939832 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:52.939807    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:11:53.100873 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:53.100839    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11044-b8ed5a"]
Nov 03 20:11:53.107404 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:53.107368    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11044-b8ed5a"]
Nov 03 20:11:53.137223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:53.137183    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7" event={"ID":"788b5e72-bede-4a35-9a85-a1b2a0174c30","Type":"ContainerStarted","Data":"27158ff7c51359bd818e1499f418cebd71f792680b0a97a3148ca9ff65b1b3d0"}
Nov 03 20:11:53.137419 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:53.137247    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7" event={"ID":"788b5e72-bede-4a35-9a85-a1b2a0174c30","Type":"ContainerStarted","Data":"af214b04d5b8bcae2a53cc2491a52d0a58cfa7dedd68ff2d7a515db917d56712"}
Nov 03 20:11:53.440463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:53.440328    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:54.147089 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:54.147039    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11044-b8ed5a" podUID="b231f0ef-7f0f-4142-a407-b035bc407f68" containerName="test-curl-11044-b8ed5a" containerID="cri-o://c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65" gracePeriod=2
Nov 03 20:11:54.449528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:54.449411    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:54.471911 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:54.471837    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7" podStartSLOduration=2.471823653 podStartE2EDuration="2.471823653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:53.163594887 +0000 UTC m=+1895.676416669" watchObservedRunningTime="2026-11-03 20:11:54.471823653 +0000 UTC m=+1896.984645434"
Nov 03 20:11:55.155794 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:55.155762    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:55.156752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:55.156733    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:11:56.446301 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:56.446276    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11044-b8ed5a"
Nov 03 20:11:56.547351 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:56.547302    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b231f0ef-7f0f-4142-a407-b035bc407f68-kube-api-access-w4ztp\" (UniqueName: \"kubernetes.io/projected/b231f0ef-7f0f-4142-a407-b035bc407f68-kube-api-access-w4ztp\") pod \"b231f0ef-7f0f-4142-a407-b035bc407f68\" (UID: \"b231f0ef-7f0f-4142-a407-b035bc407f68\") "
Nov 03 20:11:56.551961 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:56.551932    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b231f0ef-7f0f-4142-a407-b035bc407f68-kube-api-access-w4ztp" pod "b231f0ef-7f0f-4142-a407-b035bc407f68" (UID: "b231f0ef-7f0f-4142-a407-b035bc407f68"). InnerVolumeSpecName "kube-api-access-w4ztp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:56.649795 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:56.649760    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4ztp\" (UniqueName: \"kubernetes.io/projected/b231f0ef-7f0f-4142-a407-b035bc407f68-kube-api-access-w4ztp\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:57.230769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:57.230730    5600 generic.go:356] "Generic (PLEG): container finished" podID="b231f0ef-7f0f-4142-a407-b035bc407f68" containerID="c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65" exitCode=137
Nov 03 20:11:57.245475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:57.232197    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11044-b8ed5a"
Nov 03 20:11:57.248846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:57.246031    5600 scope.go:122] "RemoveContainer" containerID="c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65"
Nov 03 20:11:57.279484 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:57.278775    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11044-b8ed5a\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="b231f0ef-7f0f-4142-a407-b035bc407f68" pod="opendatahub/test-curl-11044-b8ed5a"
Nov 03 20:11:57.279484 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:57.278900    5600 scope.go:122] "RemoveContainer" containerID="c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65"
Nov 03 20:11:57.282963 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:57.282842    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65\": container with ID starting with c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65 not found: ID does not exist" containerID="c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65"
Nov 03 20:11:57.282963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:57.282896    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65"} err="failed to get container status \"c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65\": rpc error: code = NotFound desc = could not find container \"c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65\": container with ID starting with c2f197c00a63ccc61bfa9bf76f40f7e5484ea4f938a94b8e8f1d1ed530c25c65 not found: ID does not exist"
Nov 03 20:11:57.616530 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:57.616426    5600 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11044-b8ed5a\" is forbidden: User \"system:node:ip-10-0-128-24.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-24.us-east-2.compute.internal' and this object" podUID="b231f0ef-7f0f-4142-a407-b035bc407f68" pod="opendatahub/test-curl-11044-b8ed5a"
Nov 03 20:11:57.620808 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:11:57.620777    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b231f0ef-7f0f-4142-a407-b035bc407f68" path="/var/lib/kubelet/pods/b231f0ef-7f0f-4142-a407-b035bc407f68/volumes"
Nov 03 20:11:58.821165 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:11:58.821125    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:12:08.406905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:08.406867    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"]
Nov 03 20:12:08.407332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:08.407122    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy" containerID="cri-o://5275867c5aa874046fea0fd03c632a3ccd55ba79cb5567504d3cf5c698dfe674" gracePeriod=30
Nov 03 20:12:08.413222 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:08.413186    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"]
Nov 03 20:12:08.413477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:08.413413    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy" containerID="cri-o://1de9ac353fb84b54c4a6a14127f0369cb7532a183f27b4d8dce93d666a9bb497" gracePeriod=30
Nov 03 20:12:11.200506 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.200425    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"]
Nov 03 20:12:11.201095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.200817    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr" podUID="ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c" containerName="maas-api" containerID="cri-o://1a35ef16a6d5f9895f43f2fb61c319476cb133f84eab26bd49acf89f81d308fc" gracePeriod=30
Nov 03 20:12:11.211770 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.211737    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"]
Nov 03 20:12:11.212124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.212010    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch" podUID="94221571-ec07-465a-b972-02f0a50dc05b" containerName="maas-api" containerID="cri-o://f57e1abb4b71bb441898071049d1688b1917802d338620a18d28846a97f6de9e" gracePeriod=30
Nov 03 20:12:11.223624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.223596    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"]
Nov 03 20:12:11.223876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.223829    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerName="istio-proxy" containerID="cri-o://4c9ace9af392833b150f893b682f72ede76f280d961d00f3c1a868db89ab16b2" gracePeriod=30
Nov 03 20:12:11.247454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.246930    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"]
Nov 03 20:12:11.247454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.247201    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" podUID="e65c57ce-956a-4eba-a096-19d49a17b0aa" containerName="istio-proxy" containerID="cri-o://0cf72cacee8fb0555bcda14c57f315ff1620bf8e12829458165d7a7a2431f1ae" gracePeriod=30
Nov 03 20:12:11.265234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.264644    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"]
Nov 03 20:12:11.265234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.264914    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7" podUID="788b5e72-bede-4a35-9a85-a1b2a0174c30" containerName="istio-proxy" containerID="cri-o://27158ff7c51359bd818e1499f418cebd71f792680b0a97a3148ca9ff65b1b3d0" gracePeriod=30
Nov 03 20:12:11.479852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.477608    5600 generic.go:356] "Generic (PLEG): container finished" podID="ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c" containerID="1a35ef16a6d5f9895f43f2fb61c319476cb133f84eab26bd49acf89f81d308fc" exitCode=0
Nov 03 20:12:11.479852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.477705    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr" event={"ID":"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c","Type":"ContainerDied","Data":"1a35ef16a6d5f9895f43f2fb61c319476cb133f84eab26bd49acf89f81d308fc"}
Nov 03 20:12:11.522350 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.502388    5600 generic.go:356] "Generic (PLEG): container finished" podID="94221571-ec07-465a-b972-02f0a50dc05b" containerID="f57e1abb4b71bb441898071049d1688b1917802d338620a18d28846a97f6de9e" exitCode=0
Nov 03 20:12:11.522350 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.502710    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch" event={"ID":"94221571-ec07-465a-b972-02f0a50dc05b","Type":"ContainerDied","Data":"f57e1abb4b71bb441898071049d1688b1917802d338620a18d28846a97f6de9e"}
Nov 03 20:12:11.819596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.819062    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:12:11.860954 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.860915    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94221571-ec07-465a-b972-02f0a50dc05b-kube-api-access-4dp9h\" (UniqueName: \"kubernetes.io/projected/94221571-ec07-465a-b972-02f0a50dc05b-kube-api-access-4dp9h\") pod \"94221571-ec07-465a-b972-02f0a50dc05b\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") "
Nov 03 20:12:11.861200 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.861180    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-maas-api-tls\") pod \"94221571-ec07-465a-b972-02f0a50dc05b\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") "
Nov 03 20:12:11.861329 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.861310    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-metrics-certs\") pod \"94221571-ec07-465a-b972-02f0a50dc05b\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") "
Nov 03 20:12:11.861400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.861346    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94221571-ec07-465a-b972-02f0a50dc05b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94221571-ec07-465a-b972-02f0a50dc05b-openshift-service-ca\") pod \"94221571-ec07-465a-b972-02f0a50dc05b\" (UID: \"94221571-ec07-465a-b972-02f0a50dc05b\") "
Nov 03 20:12:11.863802 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.863766    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94221571-ec07-465a-b972-02f0a50dc05b-openshift-service-ca" pod "94221571-ec07-465a-b972-02f0a50dc05b" (UID: "94221571-ec07-465a-b972-02f0a50dc05b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:11.865360 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.865331    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94221571-ec07-465a-b972-02f0a50dc05b-kube-api-access-4dp9h" pod "94221571-ec07-465a-b972-02f0a50dc05b" (UID: "94221571-ec07-465a-b972-02f0a50dc05b"). InnerVolumeSpecName "kube-api-access-4dp9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:11.866293 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.866261    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-metrics-certs" pod "94221571-ec07-465a-b972-02f0a50dc05b" (UID: "94221571-ec07-465a-b972-02f0a50dc05b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:11.871941 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.871907    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-maas-api-tls" pod "94221571-ec07-465a-b972-02f0a50dc05b" (UID: "94221571-ec07-465a-b972-02f0a50dc05b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:11.912080 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.911797    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:12:11.963425 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.963388    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-maas-api-tls\") pod \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") "
Nov 03 20:12:11.963680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.963506    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-metrics-certs\") pod \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") "
Nov 03 20:12:11.963680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.963606    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-openshift-service-ca\") pod \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") "
Nov 03 20:12:11.963820 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.963764    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-kube-api-access-qsj46\" (UniqueName: \"kubernetes.io/projected/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-kube-api-access-qsj46\") pod \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\" (UID: \"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c\") "
Nov 03 20:12:11.964264 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.964115    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-openshift-service-ca" pod "ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c" (UID: "ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:11.964708 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.964686    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dp9h\" (UniqueName: \"kubernetes.io/projected/94221571-ec07-465a-b972-02f0a50dc05b-kube-api-access-4dp9h\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:11.964854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.964839    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:11.964965 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.964952    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:11.965060 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.965048    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94221571-ec07-465a-b972-02f0a50dc05b-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:11.965183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.965168    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94221571-ec07-465a-b972-02f0a50dc05b-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:11.966633 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.966599    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-metrics-certs" pod "ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c" (UID: "ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:11.966786 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.966762    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-maas-api-tls" pod "ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c" (UID: "ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:11.966861 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:11.966837    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-kube-api-access-qsj46" pod "ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c" (UID: "ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c"). InnerVolumeSpecName "kube-api-access-qsj46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:12.080580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.078879    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:12.080580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.078921    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:12.080580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.078939    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsj46\" (UniqueName: \"kubernetes.io/projected/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c-kube-api-access-qsj46\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:12.533456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.533290    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr" event={"ID":"ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c","Type":"ContainerDied","Data":"8905113c7591af229f3ed9200573645f692abfa0f9c8b28d8d24723e5ba18083"}
Nov 03 20:12:12.533456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.533369    5600 scope.go:122] "RemoveContainer" containerID="1a35ef16a6d5f9895f43f2fb61c319476cb133f84eab26bd49acf89f81d308fc"
Nov 03 20:12:12.534030 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.533555    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"
Nov 03 20:12:12.563044 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.544141    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch" event={"ID":"94221571-ec07-465a-b972-02f0a50dc05b","Type":"ContainerDied","Data":"db18c1e6448a4286de1bc4e5def1a18036b741e65ad9c596f9d8d263ffdfc8ee"}
Nov 03 20:12:12.563044 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.544260    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"
Nov 03 20:12:12.583644 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.578542    5600 scope.go:122] "RemoveContainer" containerID="f57e1abb4b71bb441898071049d1688b1917802d338620a18d28846a97f6de9e"
Nov 03 20:12:12.649756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.648784    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"]
Nov 03 20:12:12.716462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.708687    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-433102-67cd5c5694-ns8dr"]
Nov 03 20:12:12.747990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.745549    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"]
Nov 03 20:12:12.752486 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:12.752114    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-5skch"]
Nov 03 20:12:13.555646 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.555608    5600 generic.go:356] "Generic (PLEG): container finished" podID="aeded9b3-ac9b-4390-81db-6001a219415f" containerID="5275867c5aa874046fea0fd03c632a3ccd55ba79cb5567504d3cf5c698dfe674" exitCode=0
Nov 03 20:12:13.556149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.555683    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" event={"ID":"aeded9b3-ac9b-4390-81db-6001a219415f","Type":"ContainerDied","Data":"5275867c5aa874046fea0fd03c632a3ccd55ba79cb5567504d3cf5c698dfe674"}
Nov 03 20:12:13.561073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.561039    5600 generic.go:356] "Generic (PLEG): container finished" podID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerID="1de9ac353fb84b54c4a6a14127f0369cb7532a183f27b4d8dce93d666a9bb497" exitCode=0
Nov 03 20:12:13.561224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.561172    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" event={"ID":"436453e2-622e-4a77-aa2e-20c7dc39f6ad","Type":"ContainerDied","Data":"1de9ac353fb84b54c4a6a14127f0369cb7532a183f27b4d8dce93d666a9bb497"}
Nov 03 20:12:13.652040 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.652003    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94221571-ec07-465a-b972-02f0a50dc05b" path="/var/lib/kubelet/pods/94221571-ec07-465a-b972-02f0a50dc05b/volumes"
Nov 03 20:12:13.652713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.652694    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c" path="/var/lib/kubelet/pods/ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c/volumes"
Nov 03 20:12:13.790452 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.790400    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:12:13.833494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.833119    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:12:13.872189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.872144    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-certs\") pod \"aeded9b3-ac9b-4390-81db-6001a219415f\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") "
Nov 03 20:12:13.872552 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.872527    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeded9b3-ac9b-4390-81db-6001a219415f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeded9b3-ac9b-4390-81db-6001a219415f-istiod-ca-cert\") pod \"aeded9b3-ac9b-4390-81db-6001a219415f\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") "
Nov 03 20:12:13.873850 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.872986    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeded9b3-ac9b-4390-81db-6001a219415f-istiod-ca-cert" pod "aeded9b3-ac9b-4390-81db-6001a219415f" (UID: "aeded9b3-ac9b-4390-81db-6001a219415f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:13.874156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.874131    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-credential-socket\") pod \"aeded9b3-ac9b-4390-81db-6001a219415f\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") "
Nov 03 20:12:13.874368 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.874348    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-data\") pod \"aeded9b3-ac9b-4390-81db-6001a219415f\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") "
Nov 03 20:12:13.874733 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.874703    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-envoy\") pod \"aeded9b3-ac9b-4390-81db-6001a219415f\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") "
Nov 03 20:12:13.878032 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.875067    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-socket\") pod \"aeded9b3-ac9b-4390-81db-6001a219415f\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") "
Nov 03 20:12:13.883518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.878467    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-kube-api-access-b2pcv\" (UniqueName: \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-kube-api-access-b2pcv\") pod \"aeded9b3-ac9b-4390-81db-6001a219415f\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") "
Nov 03 20:12:13.883726 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.883706    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aeded9b3-ac9b-4390-81db-6001a219415f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeded9b3-ac9b-4390-81db-6001a219415f-istio-podinfo\") pod \"aeded9b3-ac9b-4390-81db-6001a219415f\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") "
Nov 03 20:12:13.893503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.883855    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-istio-token\" (UniqueName: \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-istio-token\") pod \"aeded9b3-ac9b-4390-81db-6001a219415f\" (UID: \"aeded9b3-ac9b-4390-81db-6001a219415f\") "
Nov 03 20:12:13.893503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.888413    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeded9b3-ac9b-4390-81db-6001a219415f-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:13.893503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.876701    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-credential-socket" pod "aeded9b3-ac9b-4390-81db-6001a219415f" (UID: "aeded9b3-ac9b-4390-81db-6001a219415f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:13.893503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.877689    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-data" pod "aeded9b3-ac9b-4390-81db-6001a219415f" (UID: "aeded9b3-ac9b-4390-81db-6001a219415f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:13.893503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.878146    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-socket" pod "aeded9b3-ac9b-4390-81db-6001a219415f" (UID: "aeded9b3-ac9b-4390-81db-6001a219415f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:13.893503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.878312    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-certs" pod "aeded9b3-ac9b-4390-81db-6001a219415f" (UID: "aeded9b3-ac9b-4390-81db-6001a219415f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:13.893503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.882222    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-envoy" pod "aeded9b3-ac9b-4390-81db-6001a219415f" (UID: "aeded9b3-ac9b-4390-81db-6001a219415f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:13.893503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.888886    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aeded9b3-ac9b-4390-81db-6001a219415f-istio-podinfo" pod "aeded9b3-ac9b-4390-81db-6001a219415f" (UID: "aeded9b3-ac9b-4390-81db-6001a219415f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:13.893503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.888986    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-kube-api-access-b2pcv" pod "aeded9b3-ac9b-4390-81db-6001a219415f" (UID: "aeded9b3-ac9b-4390-81db-6001a219415f"). InnerVolumeSpecName "kube-api-access-b2pcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:13.893503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:13.890419    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-istio-token" pod "aeded9b3-ac9b-4390-81db-6001a219415f" (UID: "aeded9b3-ac9b-4390-81db-6001a219415f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:14.004943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.004851    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istiod-ca-cert\") pod \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") "
Nov 03 20:12:14.008464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.005214    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-envoy\") pod \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") "
Nov 03 20:12:14.020630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.020594    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-token\" (UniqueName: \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-token\") pod \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") "
Nov 03 20:12:14.020897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.020871    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istiod-ca-cert" pod "436453e2-622e-4a77-aa2e-20c7dc39f6ad" (UID: "436453e2-622e-4a77-aa2e-20c7dc39f6ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:14.021041 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.020965    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-data\") pod \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") "
Nov 03 20:12:14.021267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.021246    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-socket\") pod \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") "
Nov 03 20:12:14.021372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.021325    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-certs\") pod \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") "
Nov 03 20:12:14.021477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.021427    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-kube-api-access-77mr4\" (UniqueName: \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-kube-api-access-77mr4\") pod \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") "
Nov 03 20:12:14.021477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.021459    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-data" pod "436453e2-622e-4a77-aa2e-20c7dc39f6ad" (UID: "436453e2-622e-4a77-aa2e-20c7dc39f6ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.021614 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.021581    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-podinfo\") pod \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") "
Nov 03 20:12:14.021700 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.021640    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-credential-socket\") pod \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\" (UID: \"436453e2-622e-4a77-aa2e-20c7dc39f6ad\") "
Nov 03 20:12:14.023445 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.022513    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-socket" pod "436453e2-622e-4a77-aa2e-20c7dc39f6ad" (UID: "436453e2-622e-4a77-aa2e-20c7dc39f6ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.023547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.023170    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-credential-socket" pod "436453e2-622e-4a77-aa2e-20c7dc39f6ad" (UID: "436453e2-622e-4a77-aa2e-20c7dc39f6ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.023547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.023306    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-certs" pod "436453e2-622e-4a77-aa2e-20c7dc39f6ad" (UID: "436453e2-622e-4a77-aa2e-20c7dc39f6ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025286    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025317    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025332    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025346    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025360    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025374    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025387    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025402    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025417    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2pcv\" (UniqueName: \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-kube-api-access-b2pcv\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025476    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025490    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeded9b3-ac9b-4390-81db-6001a219415f-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025499    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeded9b3-ac9b-4390-81db-6001a219415f-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025508    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeded9b3-ac9b-4390-81db-6001a219415f-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025523    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-token" pod "436453e2-622e-4a77-aa2e-20c7dc39f6ad" (UID: "436453e2-622e-4a77-aa2e-20c7dc39f6ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.025551    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-kube-api-access-77mr4" pod "436453e2-622e-4a77-aa2e-20c7dc39f6ad" (UID: "436453e2-622e-4a77-aa2e-20c7dc39f6ad"). InnerVolumeSpecName "kube-api-access-77mr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:14.027295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.026823    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-envoy" pod "436453e2-622e-4a77-aa2e-20c7dc39f6ad" (UID: "436453e2-622e-4a77-aa2e-20c7dc39f6ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.027999 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.027424    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-podinfo" pod "436453e2-622e-4a77-aa2e-20c7dc39f6ad" (UID: "436453e2-622e-4a77-aa2e-20c7dc39f6ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:14.127075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.127041    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.127075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.127076    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77mr4\" (UniqueName: \"kubernetes.io/projected/436453e2-622e-4a77-aa2e-20c7dc39f6ad-kube-api-access-77mr4\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.127274 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.127092    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.127274 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.127106    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436453e2-622e-4a77-aa2e-20c7dc39f6ad-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.573597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.573567    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"
Nov 03 20:12:14.577563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.577524    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx" event={"ID":"aeded9b3-ac9b-4390-81db-6001a219415f","Type":"ContainerDied","Data":"7539b947d77973df3c6167a7c1e08b720ccac729a5676b34d6bb1a9e67a321b3"}
Nov 03 20:12:14.577720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.577605    5600 scope.go:122] "RemoveContainer" containerID="5275867c5aa874046fea0fd03c632a3ccd55ba79cb5567504d3cf5c698dfe674"
Nov 03 20:12:14.582374 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.582304    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl" event={"ID":"436453e2-622e-4a77-aa2e-20c7dc39f6ad","Type":"ContainerDied","Data":"70e1d7ad4c61c9e5f8962123811a5774802c79bd159810d907b6e70a89be6e4f"}
Nov 03 20:12:14.582872 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.582848    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"
Nov 03 20:12:14.592601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.592580    5600 scope.go:122] "RemoveContainer" containerID="1de9ac353fb84b54c4a6a14127f0369cb7532a183f27b4d8dce93d666a9bb497"
Nov 03 20:12:14.610271 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.610235    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"]
Nov 03 20:12:14.621449 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.621408    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-frfbx"]
Nov 03 20:12:14.638360 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.638320    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"]
Nov 03 20:12:14.642483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:14.642462    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-84mkl"]
Nov 03 20:12:15.619077 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:15.619040    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" path="/var/lib/kubelet/pods/436453e2-622e-4a77-aa2e-20c7dc39f6ad/volumes"
Nov 03 20:12:15.620011 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:15.619989    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" path="/var/lib/kubelet/pods/aeded9b3-ac9b-4390-81db-6001a219415f/volumes"
Nov 03 20:12:17.142280 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.142230    5600 generic.go:356] "Generic (PLEG): container finished" podID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerID="4c9ace9af392833b150f893b682f72ede76f280d961d00f3c1a868db89ab16b2" exitCode=0
Nov 03 20:12:17.142893 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.142569    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" event={"ID":"17c3f78a-9dc4-463d-aa67-3d634d6caa75","Type":"ContainerDied","Data":"4c9ace9af392833b150f893b682f72ede76f280d961d00f3c1a868db89ab16b2"}
Nov 03 20:12:17.152469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.150828    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:12:17.162511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.162102    5600 generic.go:356] "Generic (PLEG): container finished" podID="e65c57ce-956a-4eba-a096-19d49a17b0aa" containerID="0cf72cacee8fb0555bcda14c57f315ff1620bf8e12829458165d7a7a2431f1ae" exitCode=0
Nov 03 20:12:17.162511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.162245    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" event={"ID":"e65c57ce-956a-4eba-a096-19d49a17b0aa","Type":"ContainerDied","Data":"0cf72cacee8fb0555bcda14c57f315ff1620bf8e12829458165d7a7a2431f1ae"}
Nov 03 20:12:17.164224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.164199    5600 generic.go:356] "Generic (PLEG): container finished" podID="788b5e72-bede-4a35-9a85-a1b2a0174c30" containerID="27158ff7c51359bd818e1499f418cebd71f792680b0a97a3148ca9ff65b1b3d0" exitCode=0
Nov 03 20:12:17.164352 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.164256    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7" event={"ID":"788b5e72-bede-4a35-9a85-a1b2a0174c30","Type":"ContainerDied","Data":"27158ff7c51359bd818e1499f418cebd71f792680b0a97a3148ca9ff65b1b3d0"}
Nov 03 20:12:17.164777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.164755    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:12:17.170062 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.170040    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:12:17.210458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.203777    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-certs\") pod \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") "
Nov 03 20:12:17.210458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.203889    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-envoy\") pod \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") "
Nov 03 20:12:17.210458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.203971    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-podinfo\") pod \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") "
Nov 03 20:12:17.210458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.204091    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-credential-socket\") pod \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") "
Nov 03 20:12:17.210458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.204164    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-token\" (UniqueName: \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-token\") pod \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") "
Nov 03 20:12:17.210458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.204217    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-socket\") pod \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") "
Nov 03 20:12:17.210458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.204267    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-kube-api-access-qvk7g\" (UniqueName: \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-kube-api-access-qvk7g\") pod \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") "
Nov 03 20:12:17.210458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.204424    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-data\") pod \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") "
Nov 03 20:12:17.210458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.204475    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istiod-ca-cert\") pod \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\" (UID: \"17c3f78a-9dc4-463d-aa67-3d634d6caa75\") "
Nov 03 20:12:17.228595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.215370    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istiod-ca-cert" pod "17c3f78a-9dc4-463d-aa67-3d634d6caa75" (UID: "17c3f78a-9dc4-463d-aa67-3d634d6caa75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:17.228595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.215644    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-socket" pod "17c3f78a-9dc4-463d-aa67-3d634d6caa75" (UID: "17c3f78a-9dc4-463d-aa67-3d634d6caa75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.228595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.219625    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-credential-socket" pod "17c3f78a-9dc4-463d-aa67-3d634d6caa75" (UID: "17c3f78a-9dc4-463d-aa67-3d634d6caa75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.228595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.221282    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-certs" pod "17c3f78a-9dc4-463d-aa67-3d634d6caa75" (UID: "17c3f78a-9dc4-463d-aa67-3d634d6caa75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.228595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.225005    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-data" pod "17c3f78a-9dc4-463d-aa67-3d634d6caa75" (UID: "17c3f78a-9dc4-463d-aa67-3d634d6caa75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.243308    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"]
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.249662    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-kube-api-access-qvk7g" pod "17c3f78a-9dc4-463d-aa67-3d634d6caa75" (UID: "17c3f78a-9dc4-463d-aa67-3d634d6caa75"). InnerVolumeSpecName "kube-api-access-qvk7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.250029    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-token" pod "17c3f78a-9dc4-463d-aa67-3d634d6caa75" (UID: "17c3f78a-9dc4-463d-aa67-3d634d6caa75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.253902    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c" containerName="maas-api"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.253922    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c" containerName="maas-api"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.253951    5600 cpu_manager.go:414] "containerMap: removing container" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.253961    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.253991    5600 cpu_manager.go:414] "containerMap: removing container" podUID="94221571-ec07-465a-b972-02f0a50dc05b" containerName="maas-api"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.253998    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94221571-ec07-465a-b972-02f0a50dc05b" containerName="maas-api"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254014    5600 cpu_manager.go:414] "containerMap: removing container" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254022    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254054    5600 cpu_manager.go:414] "containerMap: removing container" podUID="e65c57ce-956a-4eba-a096-19d49a17b0aa" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254061    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65c57ce-956a-4eba-a096-19d49a17b0aa" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254071    5600 cpu_manager.go:414] "containerMap: removing container" podUID="788b5e72-bede-4a35-9a85-a1b2a0174c30" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254078    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788b5e72-bede-4a35-9a85-a1b2a0174c30" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254093    5600 cpu_manager.go:414] "containerMap: removing container" podUID="b231f0ef-7f0f-4142-a407-b035bc407f68" containerName="test-curl-11044-b8ed5a"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254100    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b231f0ef-7f0f-4142-a407-b035bc407f68" containerName="test-curl-11044-b8ed5a"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254119    5600 cpu_manager.go:414] "containerMap: removing container" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254126    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254320    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="94221571-ec07-465a-b972-02f0a50dc05b" containerName="maas-api"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254340    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="436453e2-622e-4a77-aa2e-20c7dc39f6ad" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254355    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65c57ce-956a-4eba-a096-19d49a17b0aa" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254364    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="788b5e72-bede-4a35-9a85-a1b2a0174c30" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254375    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="b231f0ef-7f0f-4142-a407-b035bc407f68" containerName="test-curl-11044-b8ed5a"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254385    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" containerName="istio-proxy"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254399    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe46c2c-73de-49c5-b8c4-cb69c7b8cc1c" containerName="maas-api"
Nov 03 20:12:17.257358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.254409    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeded9b3-ac9b-4390-81db-6001a219415f" containerName="istio-proxy"
Nov 03 20:12:17.258863 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.258069    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-podinfo" pod "17c3f78a-9dc4-463d-aa67-3d634d6caa75" (UID: "17c3f78a-9dc4-463d-aa67-3d634d6caa75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:17.263352 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.263318    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-envoy" pod "17c3f78a-9dc4-463d-aa67-3d634d6caa75" (UID: "17c3f78a-9dc4-463d-aa67-3d634d6caa75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.263850 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.263829    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"]
Nov 03 20:12:17.264096 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.264081    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.273780 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.272589    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"]
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.311835    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-token\" (UniqueName: \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-token\") pod \"e65c57ce-956a-4eba-a096-19d49a17b0aa\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.311942    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-certs\") pod \"e65c57ce-956a-4eba-a096-19d49a17b0aa\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312008    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-podinfo\") pod \"788b5e72-bede-4a35-9a85-a1b2a0174c30\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312141    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-kube-api-access-2f56d\" (UniqueName: \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-kube-api-access-2f56d\") pod \"788b5e72-bede-4a35-9a85-a1b2a0174c30\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312188    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-data\") pod \"788b5e72-bede-4a35-9a85-a1b2a0174c30\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312315    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-certs\") pod \"788b5e72-bede-4a35-9a85-a1b2a0174c30\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312345    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-envoy\") pod \"e65c57ce-956a-4eba-a096-19d49a17b0aa\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312413    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-kube-api-access-fsc74\" (UniqueName: \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-kube-api-access-fsc74\") pod \"e65c57ce-956a-4eba-a096-19d49a17b0aa\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312475    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-socket\") pod \"788b5e72-bede-4a35-9a85-a1b2a0174c30\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312512    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-socket\") pod \"e65c57ce-956a-4eba-a096-19d49a17b0aa\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312573    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-credential-socket\") pod \"e65c57ce-956a-4eba-a096-19d49a17b0aa\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312633    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788b5e72-bede-4a35-9a85-a1b2a0174c30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788b5e72-bede-4a35-9a85-a1b2a0174c30-istiod-ca-cert\") pod \"788b5e72-bede-4a35-9a85-a1b2a0174c30\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312680    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"]
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312708    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65c57ce-956a-4eba-a096-19d49a17b0aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65c57ce-956a-4eba-a096-19d49a17b0aa-istiod-ca-cert\") pod \"e65c57ce-956a-4eba-a096-19d49a17b0aa\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312746    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-data\") pod \"e65c57ce-956a-4eba-a096-19d49a17b0aa\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312798    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312815    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-token\" (UniqueName: \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-token\") pod \"788b5e72-bede-4a35-9a85-a1b2a0174c30\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.312875    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-podinfo\") pod \"e65c57ce-956a-4eba-a096-19d49a17b0aa\" (UID: \"e65c57ce-956a-4eba-a096-19d49a17b0aa\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.313023    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-envoy\") pod \"788b5e72-bede-4a35-9a85-a1b2a0174c30\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.313065    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-credential-socket\") pod \"788b5e72-bede-4a35-9a85-a1b2a0174c30\" (UID: \"788b5e72-bede-4a35-9a85-a1b2a0174c30\") "
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.313553    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-socket" pod "788b5e72-bede-4a35-9a85-a1b2a0174c30" (UID: "788b5e72-bede-4a35-9a85-a1b2a0174c30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.313871    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-credential-socket" pod "788b5e72-bede-4a35-9a85-a1b2a0174c30" (UID: "788b5e72-bede-4a35-9a85-a1b2a0174c30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314169    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-socket" pod "e65c57ce-956a-4eba-a096-19d49a17b0aa" (UID: "e65c57ce-956a-4eba-a096-19d49a17b0aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314231    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314253    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314270    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvk7g\" (UniqueName: \"kubernetes.io/projected/17c3f78a-9dc4-463d-aa67-3d634d6caa75-kube-api-access-qvk7g\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314302    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314317    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314332    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314368    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314381    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314394    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c3f78a-9dc4-463d-aa67-3d634d6caa75-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314407    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c3f78a-9dc4-463d-aa67-3d634d6caa75-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.314674    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.315095    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-credential-socket" pod "e65c57ce-956a-4eba-a096-19d49a17b0aa" (UID: "e65c57ce-956a-4eba-a096-19d49a17b0aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.315446    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788b5e72-bede-4a35-9a85-a1b2a0174c30-istiod-ca-cert" pod "788b5e72-bede-4a35-9a85-a1b2a0174c30" (UID: "788b5e72-bede-4a35-9a85-a1b2a0174c30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.316295    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65c57ce-956a-4eba-a096-19d49a17b0aa-istiod-ca-cert" pod "e65c57ce-956a-4eba-a096-19d49a17b0aa" (UID: "e65c57ce-956a-4eba-a096-19d49a17b0aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.316575    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-data" pod "e65c57ce-956a-4eba-a096-19d49a17b0aa" (UID: "e65c57ce-956a-4eba-a096-19d49a17b0aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.321294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.321184    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-token" pod "788b5e72-bede-4a35-9a85-a1b2a0174c30" (UID: "788b5e72-bede-4a35-9a85-a1b2a0174c30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:17.334920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.332328    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"]
Nov 03 20:12:17.334920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.333577    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-envoy" pod "788b5e72-bede-4a35-9a85-a1b2a0174c30" (UID: "788b5e72-bede-4a35-9a85-a1b2a0174c30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.339703 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.336882    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-data" pod "788b5e72-bede-4a35-9a85-a1b2a0174c30" (UID: "788b5e72-bede-4a35-9a85-a1b2a0174c30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.339703 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.337088    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-certs" pod "e65c57ce-956a-4eba-a096-19d49a17b0aa" (UID: "e65c57ce-956a-4eba-a096-19d49a17b0aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.340571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.340396    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.341150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.341131    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-certs" pod "788b5e72-bede-4a35-9a85-a1b2a0174c30" (UID: "788b5e72-bede-4a35-9a85-a1b2a0174c30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.341291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.341276    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"]
Nov 03 20:12:17.349754 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.349728    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-envoy" pod "e65c57ce-956a-4eba-a096-19d49a17b0aa" (UID: "e65c57ce-956a-4eba-a096-19d49a17b0aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:17.350063 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.350035    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-token" pod "e65c57ce-956a-4eba-a096-19d49a17b0aa" (UID: "e65c57ce-956a-4eba-a096-19d49a17b0aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:17.351628 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.351598    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-kube-api-access-fsc74" pod "e65c57ce-956a-4eba-a096-19d49a17b0aa" (UID: "e65c57ce-956a-4eba-a096-19d49a17b0aa"). InnerVolumeSpecName "kube-api-access-fsc74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:17.352064 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.352039    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-podinfo" pod "788b5e72-bede-4a35-9a85-a1b2a0174c30" (UID: "788b5e72-bede-4a35-9a85-a1b2a0174c30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:17.355134 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.350040    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-podinfo" pod "e65c57ce-956a-4eba-a096-19d49a17b0aa" (UID: "e65c57ce-956a-4eba-a096-19d49a17b0aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:17.359502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.356195    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-kube-api-access-2f56d" pod "788b5e72-bede-4a35-9a85-a1b2a0174c30" (UID: "788b5e72-bede-4a35-9a85-a1b2a0174c30"). InnerVolumeSpecName "kube-api-access-2f56d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:17.416505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416463    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s76l9\" (UniqueName: \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-kube-api-access-s76l9\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.416689 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416514    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vlvm\" (UniqueName: \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-kube-api-access-6vlvm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.416689 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416564    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.416689 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416594    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.416689 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416628    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.416689 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416661    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gq4q\" (UniqueName: \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-kube-api-access-9gq4q\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.416971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416693    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-token\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.416971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416723    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.416971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416786    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.416971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416817    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.416971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416842    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.416971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416881    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istiod-ca-cert\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.416971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416926    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-credential-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.417304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.416988    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-envoy\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.417304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417021    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.417304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417047    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-certs\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.417304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417145    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.417304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417169    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.417304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417195    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-data\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.417304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417224    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.417304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417261    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417318    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417365    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417408    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417506    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-podinfo\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417592    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417608    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788b5e72-bede-4a35-9a85-a1b2a0174c30-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417621    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65c57ce-956a-4eba-a096-19d49a17b0aa-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417635    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417650    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417663    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417678    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417691    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417705    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417718    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417733    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f56d\" (UniqueName: \"kubernetes.io/projected/788b5e72-bede-4a35-9a85-a1b2a0174c30-kube-api-access-2f56d\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.417752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417747    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.418535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417764    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788b5e72-bede-4a35-9a85-a1b2a0174c30-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.418535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417779    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.418535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417793    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsc74\" (UniqueName: \"kubernetes.io/projected/e65c57ce-956a-4eba-a096-19d49a17b0aa-kube-api-access-fsc74\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.418535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.417808    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c57ce-956a-4eba-a096-19d49a17b0aa-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:17.519238 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519189    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-token\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.519507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519252    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.519507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519326    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.519507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519359    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.519507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519387    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.519507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519448    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istiod-ca-cert\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.519507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519505    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-credential-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.519776 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519570    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-envoy\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.519776 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519603    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.519776 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519630    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-certs\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.519776 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519727    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.519776 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519753    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.519968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519779    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-data\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.519968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519808    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.519968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519843    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.519968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519904    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.519968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519951    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.520120 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.519977    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.520120 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.520050    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-podinfo\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.520178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.520114    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s76l9\" (UniqueName: \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-kube-api-access-s76l9\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.520178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.520141    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vlvm\" (UniqueName: \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-kube-api-access-6vlvm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.520239 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.520200    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.520272 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.520233    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.520310 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.520291    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.520349 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.520333    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gq4q\" (UniqueName: \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-kube-api-access-9gq4q\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.521885 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.521853    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.522280 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.522255    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.522621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.522589    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istiod-ca-cert\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.522868 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.522847    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-certs\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.522977 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.522863    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.524292 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.524262    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.525131 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.525105    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.525849 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.525785    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-credential-socket\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.527241 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.527204    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-data\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.527949 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.527917    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.532862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.532819    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.536146 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.536113    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.537482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.537108    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.542669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.538589    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-token\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.542669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.540172    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gq4q\" (UniqueName: \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-kube-api-access-9gq4q\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.542669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.540635    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.542669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.540951    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s76l9\" (UniqueName: \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-kube-api-access-s76l9\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.542669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.541103    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.542669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.541761    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-envoy\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.542669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.542138    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.542669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.542326    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.542669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.542479    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vlvm\" (UniqueName: \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-kube-api-access-6vlvm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.545534 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.545481    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.553133 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.551204    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9cmmw\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.553133 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.553085    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-podinfo\") pod \"e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") " pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:17.685911 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.685877    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:17.715162 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.715091    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:17.748682 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:17.748383    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:18.003573 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.003543    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"]
Nov 03 20:12:18.008575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.008539    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:18.008668 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.008602    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:18.039930 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.039876    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"]
Nov 03 20:12:18.047422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.047376    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:18.047551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.047493    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:18.047551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.047539    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:18.108411 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.104327    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"]
Nov 03 20:12:18.108411 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.108045    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:18.108411 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.108174    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:18.108411 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.108247    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:18.178528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.177947    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll" event={"ID":"e65c57ce-956a-4eba-a096-19d49a17b0aa","Type":"ContainerDied","Data":"c086f6bd16c718e6f6f35219f773350393c88b12cba965d0c57fd7f11a49b370"}
Nov 03 20:12:18.178528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.178003    5600 scope.go:122] "RemoveContainer" containerID="0cf72cacee8fb0555bcda14c57f315ff1620bf8e12829458165d7a7a2431f1ae"
Nov 03 20:12:18.178528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.178111    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"
Nov 03 20:12:18.183479 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.181464    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7" event={"ID":"788b5e72-bede-4a35-9a85-a1b2a0174c30","Type":"ContainerDied","Data":"af214b04d5b8bcae2a53cc2491a52d0a58cfa7dedd68ff2d7a515db917d56712"}
Nov 03 20:12:18.183479 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.181556    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"
Nov 03 20:12:18.185278 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.185255    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" event={"ID":"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035","Type":"ContainerStarted","Data":"da4b3268092b4acc0664c3a902099a92426cb6397f95d4ce433b35946a004d7b"}
Nov 03 20:12:18.188152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.188130    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz" event={"ID":"17c3f78a-9dc4-463d-aa67-3d634d6caa75","Type":"ContainerDied","Data":"e9edfe949cb511e1b9a72e44785e8da3ad8568998e2ccabc5010be7020dcaf1f"}
Nov 03 20:12:18.188260 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.188224    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"
Nov 03 20:12:18.192533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.192505    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw" event={"ID":"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25","Type":"ContainerStarted","Data":"48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca"}
Nov 03 20:12:18.192638 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.192546    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw" event={"ID":"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25","Type":"ContainerStarted","Data":"a194982e36dd8585ed9be944f684a0c17ee6409cb945d576897f0780ac273735"}
Nov 03 20:12:18.193159 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.193137    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:18.198701 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.196025    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" event={"ID":"aec08d5e-0c0e-459e-85b4-0b80b5b60b35","Type":"ContainerStarted","Data":"fecf13b430e4f6326fb818edba8bb3cfaeafd5d4d368ee4887e384d1472b2b0a"}
Nov 03 20:12:18.212375 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.212319    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" podStartSLOduration=7.212285073 podStartE2EDuration="7.212285073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:18.210929507 +0000 UTC m=+1920.723751290" watchObservedRunningTime="2026-11-03 20:12:18.212285073 +0000 UTC m=+1920.725106859"
Nov 03 20:12:18.265370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.254842    5600 scope.go:122] "RemoveContainer" containerID="27158ff7c51359bd818e1499f418cebd71f792680b0a97a3148ca9ff65b1b3d0"
Nov 03 20:12:18.266334 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.266310    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"]
Nov 03 20:12:18.278932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.278903    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-jczrz"]
Nov 03 20:12:18.279920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.279877    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw" podStartSLOduration=7.279865829 podStartE2EDuration="7.279865829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:18.25135724 +0000 UTC m=+1920.764179017" watchObservedRunningTime="2026-11-03 20:12:18.279865829 +0000 UTC m=+1920.792687610"
Nov 03 20:12:18.301863 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.300251    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"]
Nov 03 20:12:18.309730 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.309688    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-vqvll"]
Nov 03 20:12:18.362362 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.344526    5600 scope.go:122] "RemoveContainer" containerID="4c9ace9af392833b150f893b682f72ede76f280d961d00f3c1a868db89ab16b2"
Nov 03 20:12:18.362362 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.345703    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"]
Nov 03 20:12:18.362362 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.354381    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-7bjv7"]
Nov 03 20:12:18.598121 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.597051    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"]
Nov 03 20:12:18.598121 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.597319    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n" podUID="84f6045e-e98a-4f1e-8208-cff2804bd400" containerName="maas-api" containerID="cri-o://d1f88939a6af4ce62c32c66efebd97675ea86b3baff682e1b67ff11d0278e6ed" gracePeriod=30
Nov 03 20:12:18.752691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:18.752606    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:19.380403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.379461    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" event={"ID":"aec08d5e-0c0e-459e-85b4-0b80b5b60b35","Type":"ContainerStarted","Data":"1e37cbfa6f62d5c14de64e1b710182d24fa494a379f0632b2389f04fb5905a15"}
Nov 03 20:12:19.406731 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.406693    5600 generic.go:356] "Generic (PLEG): container finished" podID="84f6045e-e98a-4f1e-8208-cff2804bd400" containerID="d1f88939a6af4ce62c32c66efebd97675ea86b3baff682e1b67ff11d0278e6ed" exitCode=0
Nov 03 20:12:19.407069 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.407045    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n" event={"ID":"84f6045e-e98a-4f1e-8208-cff2804bd400","Type":"ContainerDied","Data":"d1f88939a6af4ce62c32c66efebd97675ea86b3baff682e1b67ff11d0278e6ed"}
Nov 03 20:12:19.422899 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.422865    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" event={"ID":"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035","Type":"ContainerStarted","Data":"a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346"}
Nov 03 20:12:19.436704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.436668    5600 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9cmmw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:12:19.436984 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.436958    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw" podUID="026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:12:19.887087 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.887050    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c3f78a-9dc4-463d-aa67-3d634d6caa75" path="/var/lib/kubelet/pods/17c3f78a-9dc4-463d-aa67-3d634d6caa75/volumes"
Nov 03 20:12:19.889693 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.888231    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788b5e72-bede-4a35-9a85-a1b2a0174c30" path="/var/lib/kubelet/pods/788b5e72-bede-4a35-9a85-a1b2a0174c30/volumes"
Nov 03 20:12:19.917120 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.917087    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65c57ce-956a-4eba-a096-19d49a17b0aa" path="/var/lib/kubelet/pods/e65c57ce-956a-4eba-a096-19d49a17b0aa/volumes"
Nov 03 20:12:19.946741 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.917930    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:19.948237 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.889919    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:12:19.948546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.948496    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:12:19.965518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:19.962334    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:20.199981 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.173407    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" podStartSLOduration=9.173383417 podStartE2EDuration="9.173383417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:19.436905434 +0000 UTC m=+1921.949727217" watchObservedRunningTime="2026-11-03 20:12:20.173383417 +0000 UTC m=+1922.686205201"
Nov 03 20:12:20.262530 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.259758    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:12:20.350535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.347284    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-maas-api-tls\") pod \"84f6045e-e98a-4f1e-8208-cff2804bd400\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") "
Nov 03 20:12:20.352348 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.351831    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f6045e-e98a-4f1e-8208-cff2804bd400-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f6045e-e98a-4f1e-8208-cff2804bd400-openshift-service-ca\") pod \"84f6045e-e98a-4f1e-8208-cff2804bd400\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") "
Nov 03 20:12:20.353381 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.353361    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f6045e-e98a-4f1e-8208-cff2804bd400-kube-api-access-fltbg\" (UniqueName: \"kubernetes.io/projected/84f6045e-e98a-4f1e-8208-cff2804bd400-kube-api-access-fltbg\") pod \"84f6045e-e98a-4f1e-8208-cff2804bd400\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") "
Nov 03 20:12:20.354860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.354188    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-metrics-certs\") pod \"84f6045e-e98a-4f1e-8208-cff2804bd400\" (UID: \"84f6045e-e98a-4f1e-8208-cff2804bd400\") "
Nov 03 20:12:20.360158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.360134    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f6045e-e98a-4f1e-8208-cff2804bd400-openshift-service-ca" pod "84f6045e-e98a-4f1e-8208-cff2804bd400" (UID: "84f6045e-e98a-4f1e-8208-cff2804bd400"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:20.366222 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.366182    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f6045e-e98a-4f1e-8208-cff2804bd400-kube-api-access-fltbg" pod "84f6045e-e98a-4f1e-8208-cff2804bd400" (UID: "84f6045e-e98a-4f1e-8208-cff2804bd400"). InnerVolumeSpecName "kube-api-access-fltbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:20.366840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.366801    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-maas-api-tls" pod "84f6045e-e98a-4f1e-8208-cff2804bd400" (UID: "84f6045e-e98a-4f1e-8208-cff2804bd400"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:20.367922 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.367903    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-metrics-certs" pod "84f6045e-e98a-4f1e-8208-cff2804bd400" (UID: "84f6045e-e98a-4f1e-8208-cff2804bd400"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:20.473251 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.473217    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:20.473782 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.473764    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84f6045e-e98a-4f1e-8208-cff2804bd400-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:20.473880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.473869    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f6045e-e98a-4f1e-8208-cff2804bd400-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:20.473952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.473942    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fltbg\" (UniqueName: \"kubernetes.io/projected/84f6045e-e98a-4f1e-8208-cff2804bd400-kube-api-access-fltbg\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:20.498963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.498933    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"
Nov 03 20:12:20.533853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.500165    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n" event={"ID":"84f6045e-e98a-4f1e-8208-cff2804bd400","Type":"ContainerDied","Data":"60308c363af7bb9c024e15f8fd1391baa78f3d0f31f0d2bba99f159c30401ead"}
Nov 03 20:12:20.535045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.534093    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:20.535324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.535307    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:12:20.535457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.534940    5600 scope.go:122] "RemoveContainer" containerID="d1f88939a6af4ce62c32c66efebd97675ea86b3baff682e1b67ff11d0278e6ed"
Nov 03 20:12:20.546526 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.535596    5600 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9cmmw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:12:20.546526 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.535645    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw" podUID="026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:12:20.649398 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.648604    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"]
Nov 03 20:12:20.669043 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.669008    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7f5c-7587d685b4-jpc6n"]
Nov 03 20:12:20.732585 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.732493    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:12:20.732854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:20.732824    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:12:21.627049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:21.627008    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f6045e-e98a-4f1e-8208-cff2804bd400" path="/var/lib/kubelet/pods/84f6045e-e98a-4f1e-8208-cff2804bd400/volumes"
Nov 03 20:12:21.717040 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:21.717001    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:12:21.717289 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:21.717261    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:12:22.542122 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:22.542092    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"]
Nov 03 20:12:22.718225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:22.718179    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:12:22.718909 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:22.718873    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:12:23.515197 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:23.515161    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:12:23.716459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:23.716120    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:12:23.716459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:23.716200    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:12:24.721045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:24.721001    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:12:24.721586 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:24.721068    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:12:25.722489 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:25.722416    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:25.723126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:25.722777    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:25.723978 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:25.723955    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:12:26.936712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:26.936676    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"]
Nov 03 20:12:26.937420 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:26.937388    5600 cpu_manager.go:414] "containerMap: removing container" podUID="84f6045e-e98a-4f1e-8208-cff2804bd400" containerName="maas-api"
Nov 03 20:12:26.937502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:26.937422    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f6045e-e98a-4f1e-8208-cff2804bd400" containerName="maas-api"
Nov 03 20:12:26.937579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:26.937568    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f6045e-e98a-4f1e-8208-cff2804bd400" containerName="maas-api"
Nov 03 20:12:26.942626 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:26.942604    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:26.947783 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:26.946279    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:12:26.947783 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:26.947690    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:12:26.968485 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:26.965487    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"]
Nov 03 20:12:27.020051 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.020013    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30019952-0426-409a-9b21-c42086a46970-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-433102-bgzzk\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:27.020248 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.020111    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30019952-0426-409a-9b21-c42086a46970-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-433102-bgzzk\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:27.020446 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.020407    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ws8k\" (UniqueName: \"kubernetes.io/projected/30019952-0426-409a-9b21-c42086a46970-kube-api-access-6ws8k\") pod \"maas-api-revoke-keys-e2e-api-a-433102-bgzzk\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:27.130459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.126864    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30019952-0426-409a-9b21-c42086a46970-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-433102-bgzzk\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:27.130459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.126982    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30019952-0426-409a-9b21-c42086a46970-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-433102-bgzzk\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:27.130459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.127239    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ws8k\" (UniqueName: \"kubernetes.io/projected/30019952-0426-409a-9b21-c42086a46970-kube-api-access-6ws8k\") pod \"maas-api-revoke-keys-e2e-api-a-433102-bgzzk\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:27.148458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.142126    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30019952-0426-409a-9b21-c42086a46970-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-433102-bgzzk\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:27.220516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.220344    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30019952-0426-409a-9b21-c42086a46970-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-433102-bgzzk\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:27.221045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.221019    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ws8k\" (UniqueName: \"kubernetes.io/projected/30019952-0426-409a-9b21-c42086a46970-kube-api-access-6ws8k\") pod \"maas-api-revoke-keys-e2e-api-a-433102-bgzzk\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:27.270070 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.269814    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:27.639959 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.639924    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"]
Nov 03 20:12:27.689513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.688905    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"]
Nov 03 20:12:27.689513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.688963    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"]
Nov 03 20:12:27.689513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.689064    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.690223 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:12:27.690191    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30019952_0426_409a_9b21_c42086a46970.slice/crio-fa60af126da176c86eecbe61a3b0b70c07c35a5210215a2031662f81630397bc WatchSource:0}: Error finding container fa60af126da176c86eecbe61a3b0b70c07c35a5210215a2031662f81630397bc: Status 404 returned error can't find the container with id fa60af126da176c86eecbe61a3b0b70c07c35a5210215a2031662f81630397bc
Nov 03 20:12:27.694156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.693728    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-663013-openshift-default-dockercfg-nk45n\""
Nov 03 20:12:27.759214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.758820    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istiod-ca-cert\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.759214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.758881    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-podinfo\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.759214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.758945    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-credential-socket\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.759214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.758980    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-envoy\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.759214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.759059    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-data\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.759214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.759199    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-socket\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.787456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.783617    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftflt\" (UniqueName: \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-kube-api-access-ftflt\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.787456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.783693    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-certs\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.787456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.783787    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-token\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.903461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.887585    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-credential-socket\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.903461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.887644    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-envoy\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.903461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.887720    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-data\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.903461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.887839    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-socket\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.903461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.887928    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftflt\" (UniqueName: \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-kube-api-access-ftflt\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.903461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.887969    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-certs\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.903461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.888051    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-token\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.903461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.888212    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istiod-ca-cert\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.903461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.888249    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-podinfo\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.903461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.901974    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-socket\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.903461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.902233    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-credential-socket\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.910135 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.910100    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-data\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.937455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.913882    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-certs\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.938759 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.929811    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istiod-ca-cert\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:27.961529 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:27.954177    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-podinfo\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:28.089579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:28.072817    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-token\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:28.089579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:28.075918    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftflt\" (UniqueName: \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-kube-api-access-ftflt\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:28.089579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:28.081483    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-envoy\") pod \"e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") " pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:28.319724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:28.319621    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:28.644458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:28.640747    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk" event={"ID":"30019952-0426-409a-9b21-c42086a46970","Type":"ContainerStarted","Data":"9ac7b6d9826f2abc2c7de0e3bb6bb15c2a3f0fe8257a4e56808b083de2d3ff99"}
Nov 03 20:12:28.644458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:28.640813    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk" event={"ID":"30019952-0426-409a-9b21-c42086a46970","Type":"ContainerStarted","Data":"fa60af126da176c86eecbe61a3b0b70c07c35a5210215a2031662f81630397bc"}
Nov 03 20:12:28.666089 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:28.666054    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"]
Nov 03 20:12:28.699701 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:28.681162    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk" podStartSLOduration=2.681141663 podStartE2EDuration="2.681141663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:28.666264638 +0000 UTC m=+1931.179086436" watchObservedRunningTime="2026-11-03 20:12:28.681141663 +0000 UTC m=+1931.193963446"
Nov 03 20:12:28.706931 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:28.701499    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:28.706931 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:28.701611    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:28.706931 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:28.701656    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:28.818902 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:12:28.818009    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:12:29.662878 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:29.662838    5600 generic.go:356] "Generic (PLEG): container finished" podID="30019952-0426-409a-9b21-c42086a46970" containerID="9ac7b6d9826f2abc2c7de0e3bb6bb15c2a3f0fe8257a4e56808b083de2d3ff99" exitCode=0
Nov 03 20:12:29.663386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:29.663025    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk" event={"ID":"30019952-0426-409a-9b21-c42086a46970","Type":"ContainerDied","Data":"9ac7b6d9826f2abc2c7de0e3bb6bb15c2a3f0fe8257a4e56808b083de2d3ff99"}
Nov 03 20:12:29.666009 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:29.665915    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" event={"ID":"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec","Type":"ContainerStarted","Data":"cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830"}
Nov 03 20:12:29.666009 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:29.665972    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" event={"ID":"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec","Type":"ContainerStarted","Data":"5d70213a35e82614b250bfdb1af3e6d4f320f900265c66ee538696d96bc4d798"}
Nov 03 20:12:29.747272 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:29.747202    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" podStartSLOduration=2.747183007 podStartE2EDuration="2.747183007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:29.726995171 +0000 UTC m=+1932.239816966" watchObservedRunningTime="2026-11-03 20:12:29.747183007 +0000 UTC m=+1932.260004790"
Nov 03 20:12:30.329459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:30.326504    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:30.380844 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:30.380810    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:30.674923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:30.674206    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:30.675275 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:30.675258    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:12:30.946542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:30.946416    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"]
Nov 03 20:12:30.951138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:30.951113    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:30.963309 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:30.962834    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"]
Nov 03 20:12:31.045413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.045380    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:31.045632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.045469    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnnw\" (UniqueName: \"kubernetes.io/projected/7c655eef-f597-485d-aa7c-543b615096c3-kube-api-access-ncnnw\") pod \"maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:31.045632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.045544    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:31.045745 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.045639    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:31.146516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.146482    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30019952-0426-409a-9b21-c42086a46970-kube-api-access-6ws8k\" (UniqueName: \"kubernetes.io/projected/30019952-0426-409a-9b21-c42086a46970-kube-api-access-6ws8k\") pod \"30019952-0426-409a-9b21-c42086a46970\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") "
Nov 03 20:12:31.146697 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.146661    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30019952-0426-409a-9b21-c42086a46970-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30019952-0426-409a-9b21-c42086a46970-maas-api-openshift-ca\") pod \"30019952-0426-409a-9b21-c42086a46970\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") "
Nov 03 20:12:31.146749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.146711    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30019952-0426-409a-9b21-c42086a46970-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30019952-0426-409a-9b21-c42086a46970-maas-api-xks-ca\") pod \"30019952-0426-409a-9b21-c42086a46970\" (UID: \"30019952-0426-409a-9b21-c42086a46970\") "
Nov 03 20:12:31.147035 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.147014    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:31.147106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.147070    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnnw\" (UniqueName: \"kubernetes.io/projected/7c655eef-f597-485d-aa7c-543b615096c3-kube-api-access-ncnnw\") pod \"maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:31.147308 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.147287    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:31.147409 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.147349    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30019952-0426-409a-9b21-c42086a46970-maas-api-openshift-ca" pod "30019952-0426-409a-9b21-c42086a46970" (UID: "30019952-0426-409a-9b21-c42086a46970"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:31.147606 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.147580    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30019952-0426-409a-9b21-c42086a46970-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:31.148310 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.148282    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:31.150614 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.150582    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:31.150832 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.150805    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30019952-0426-409a-9b21-c42086a46970-kube-api-access-6ws8k" pod "30019952-0426-409a-9b21-c42086a46970" (UID: "30019952-0426-409a-9b21-c42086a46970"). InnerVolumeSpecName "kube-api-access-6ws8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:31.151857 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.151815    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30019952-0426-409a-9b21-c42086a46970-maas-api-xks-ca" pod "30019952-0426-409a-9b21-c42086a46970" (UID: "30019952-0426-409a-9b21-c42086a46970"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:31.165849 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.165822    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnnw\" (UniqueName: \"kubernetes.io/projected/7c655eef-f597-485d-aa7c-543b615096c3-kube-api-access-ncnnw\") pod \"maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:31.249707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.249626    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ws8k\" (UniqueName: \"kubernetes.io/projected/30019952-0426-409a-9b21-c42086a46970-kube-api-access-6ws8k\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:31.249707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.249656    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30019952-0426-409a-9b21-c42086a46970-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:31.338231 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.338195    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:31.633195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.633172    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"]
Nov 03 20:12:31.635748 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:12:31.635709    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c655eef_f597_485d_aa7c_543b615096c3.slice/crio-e13fcddd51f22e62c6894f0141b8a730f43952404373806248296b215097a6dd WatchSource:0}: Error finding container e13fcddd51f22e62c6894f0141b8a730f43952404373806248296b215097a6dd: Status 404 returned error can't find the container with id e13fcddd51f22e62c6894f0141b8a730f43952404373806248296b215097a6dd
Nov 03 20:12:31.683930 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.683895    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"
Nov 03 20:12:31.684339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.683933    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk" event={"ID":"30019952-0426-409a-9b21-c42086a46970","Type":"ContainerDied","Data":"fa60af126da176c86eecbe61a3b0b70c07c35a5210215a2031662f81630397bc"}
Nov 03 20:12:31.684339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.683996    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa60af126da176c86eecbe61a3b0b70c07c35a5210215a2031662f81630397bc"
Nov 03 20:12:31.685726 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:31.685699    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76" event={"ID":"7c655eef-f597-485d-aa7c-543b615096c3","Type":"ContainerStarted","Data":"e13fcddd51f22e62c6894f0141b8a730f43952404373806248296b215097a6dd"}
Nov 03 20:12:32.755765 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:32.755532    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76" event={"ID":"7c655eef-f597-485d-aa7c-543b615096c3","Type":"ContainerStarted","Data":"fe936826800b8a8e16ed57ef3e88c99f01b19c8397407835ee3c941684274791"}
Nov 03 20:12:32.789454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:32.781823    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76" podStartSLOduration=2.781804846 podStartE2EDuration="2.781804846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:32.780164474 +0000 UTC m=+1935.292986257" watchObservedRunningTime="2026-11-03 20:12:32.781804846 +0000 UTC m=+1935.294626629"
Nov 03 20:12:33.739521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.739483    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"]
Nov 03 20:12:33.740781 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.740755    5600 cpu_manager.go:414] "containerMap: removing container" podUID="30019952-0426-409a-9b21-c42086a46970" containerName="revoke-keys"
Nov 03 20:12:33.740781 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.740781    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30019952-0426-409a-9b21-c42086a46970" containerName="revoke-keys"
Nov 03 20:12:33.740984 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.740972    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="30019952-0426-409a-9b21-c42086a46970" containerName="revoke-keys"
Nov 03 20:12:33.758753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.755305    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:12:33.759262 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.759227    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"]
Nov 03 20:12:33.769644 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.769604    5600 generic.go:356] "Generic (PLEG): container finished" podID="7c655eef-f597-485d-aa7c-543b615096c3" containerID="fe936826800b8a8e16ed57ef3e88c99f01b19c8397407835ee3c941684274791" exitCode=0
Nov 03 20:12:33.769839 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.769818    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76" event={"ID":"7c655eef-f597-485d-aa7c-543b615096c3","Type":"ContainerDied","Data":"fe936826800b8a8e16ed57ef3e88c99f01b19c8397407835ee3c941684274791"}
Nov 03 20:12:33.809811 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.809759    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b5nd\" (UniqueName: \"kubernetes.io/projected/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-kube-api-access-8b5nd\") pod \"payload-processing-e2e-inf-a-663013-848b6d9486-s696n\" (UID: \"d0ac6eff-118d-4cd7-9a34-f173552e6bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:12:33.810069 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.810047    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-663013-848b6d9486-s696n\" (UID: \"d0ac6eff-118d-4cd7-9a34-f173552e6bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:12:33.834515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.815717    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-663013\""
Nov 03 20:12:33.834515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.815847    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-663013-dockercfg-mxskt\""
Nov 03 20:12:33.919771 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.919735    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-663013-848b6d9486-s696n\" (UID: \"d0ac6eff-118d-4cd7-9a34-f173552e6bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:12:33.939528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.939125    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b5nd\" (UniqueName: \"kubernetes.io/projected/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-kube-api-access-8b5nd\") pod \"payload-processing-e2e-inf-a-663013-848b6d9486-s696n\" (UID: \"d0ac6eff-118d-4cd7-9a34-f173552e6bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:12:33.940233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.927368    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-663013-848b6d9486-s696n\" (UID: \"d0ac6eff-118d-4cd7-9a34-f173552e6bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:12:33.988454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:33.988371    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b5nd\" (UniqueName: \"kubernetes.io/projected/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-kube-api-access-8b5nd\") pod \"payload-processing-e2e-inf-a-663013-848b6d9486-s696n\" (UID: \"d0ac6eff-118d-4cd7-9a34-f173552e6bc0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:12:34.109857 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:34.109763    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:12:34.490630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:34.490597    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"]
Nov 03 20:12:34.491859 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:12:34.491831    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ac6eff_118d_4cd7_9a34_f173552e6bc0.slice/crio-e27b2831f992594ffad259751c28d41c1f9ed48c556067e305d7841153133c55 WatchSource:0}: Error finding container e27b2831f992594ffad259751c28d41c1f9ed48c556067e305d7841153133c55: Status 404 returned error can't find the container with id e27b2831f992594ffad259751c28d41c1f9ed48c556067e305d7841153133c55
Nov 03 20:12:34.778318 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:34.778217    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n" event={"ID":"d0ac6eff-118d-4cd7-9a34-f173552e6bc0","Type":"ContainerStarted","Data":"8bb30f334055c1d9522c6c907e83ae626c261c6213fb46cf7f1f8136df27ccfa"}
Nov 03 20:12:34.778318 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:34.778286    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n" event={"ID":"d0ac6eff-118d-4cd7-9a34-f173552e6bc0","Type":"ContainerStarted","Data":"e27b2831f992594ffad259751c28d41c1f9ed48c556067e305d7841153133c55"}
Nov 03 20:12:34.778318 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:34.778315    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:12:34.961526 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:34.958931    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:35.028681 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.025234    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n" podStartSLOduration=2.025215012 podStartE2EDuration="2.025215012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:34.823589842 +0000 UTC m=+1937.336411627" watchObservedRunningTime="2026-11-03 20:12:35.025215012 +0000 UTC m=+1937.538036795"
Nov 03 20:12:35.090524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.090488    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-xks-ca\") pod \"7c655eef-f597-485d-aa7c-543b615096c3\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") "
Nov 03 20:12:35.090718 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.090532    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c655eef-f597-485d-aa7c-543b615096c3-kube-api-access-ncnnw\" (UniqueName: \"kubernetes.io/projected/7c655eef-f597-485d-aa7c-543b615096c3-kube-api-access-ncnnw\") pod \"7c655eef-f597-485d-aa7c-543b615096c3\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") "
Nov 03 20:12:35.091010 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.090816    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-openshift-ca\") pod \"7c655eef-f597-485d-aa7c-543b615096c3\" (UID: \"7c655eef-f597-485d-aa7c-543b615096c3\") "
Nov 03 20:12:35.091306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.091277    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-openshift-ca" pod "7c655eef-f597-485d-aa7c-543b615096c3" (UID: "7c655eef-f597-485d-aa7c-543b615096c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:35.091792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.091717    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:35.093531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.093505    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c655eef-f597-485d-aa7c-543b615096c3-kube-api-access-ncnnw" pod "7c655eef-f597-485d-aa7c-543b615096c3" (UID: "7c655eef-f597-485d-aa7c-543b615096c3"). InnerVolumeSpecName "kube-api-access-ncnnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:35.093982 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.093859    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-xks-ca" pod "7c655eef-f597-485d-aa7c-543b615096c3" (UID: "7c655eef-f597-485d-aa7c-543b615096c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:35.194362 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.194322    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c655eef-f597-485d-aa7c-543b615096c3-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:35.194362 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.194364    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncnnw\" (UniqueName: \"kubernetes.io/projected/7c655eef-f597-485d-aa7c-543b615096c3-kube-api-access-ncnnw\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:35.790644 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.790611    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"
Nov 03 20:12:35.790644 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.790634    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76" event={"ID":"7c655eef-f597-485d-aa7c-543b615096c3","Type":"ContainerDied","Data":"e13fcddd51f22e62c6894f0141b8a730f43952404373806248296b215097a6dd"}
Nov 03 20:12:35.791113 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:35.790687    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e13fcddd51f22e62c6894f0141b8a730f43952404373806248296b215097a6dd"
Nov 03 20:12:37.711750 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:37.705514    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j" podUID="8316a979-44c7-4b82-8df4-1a1ced93d9a3" containerName="main" containerID="cri-o://3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c" gracePeriod=45
Nov 03 20:12:38.037510 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.026944    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j" podUID="8316a979-44c7-4b82-8df4-1a1ced93d9a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": dial tcp 10.134.0.90:8000: connect: connection refused"
Nov 03 20:12:38.166535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.166495    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"]
Nov 03 20:12:38.167073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.167030    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" containerID="cri-o://00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263" gracePeriod=30
Nov 03 20:12:38.488971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.488932    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"]
Nov 03 20:12:38.489422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.489375    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy" containerID="cri-o://021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f" gracePeriod=30
Nov 03 20:12:38.568333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.568303    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:12:38.645511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.644230    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-home\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-home\") pod \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") "
Nov 03 20:12:38.645511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.644356    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tmp-dir\") pod \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") "
Nov 03 20:12:38.645511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.644446    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-model-cache\") pod \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") "
Nov 03 20:12:38.645511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.644576    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tls-certs\") pod \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") "
Nov 03 20:12:38.645511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.644684    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-dshm\") pod \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") "
Nov 03 20:12:38.645511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.644713    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8316a979-44c7-4b82-8df4-1a1ced93d9a3-kube-api-access-sgj4c\" (UniqueName: \"kubernetes.io/projected/8316a979-44c7-4b82-8df4-1a1ced93d9a3-kube-api-access-sgj4c\") pod \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\" (UID: \"8316a979-44c7-4b82-8df4-1a1ced93d9a3\") "
Nov 03 20:12:38.645511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.644914    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tmp-dir" pod "8316a979-44c7-4b82-8df4-1a1ced93d9a3" (UID: "8316a979-44c7-4b82-8df4-1a1ced93d9a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:38.646235 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.645867    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-home" pod "8316a979-44c7-4b82-8df4-1a1ced93d9a3" (UID: "8316a979-44c7-4b82-8df4-1a1ced93d9a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:38.646779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.646753    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:38.646881 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.646786    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:38.646881 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.646834    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-model-cache" pod "8316a979-44c7-4b82-8df4-1a1ced93d9a3" (UID: "8316a979-44c7-4b82-8df4-1a1ced93d9a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:38.651477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.650366    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8316a979-44c7-4b82-8df4-1a1ced93d9a3-kube-api-access-sgj4c" pod "8316a979-44c7-4b82-8df4-1a1ced93d9a3" (UID: "8316a979-44c7-4b82-8df4-1a1ced93d9a3"). InnerVolumeSpecName "kube-api-access-sgj4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:38.651477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.650730    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tls-certs" pod "8316a979-44c7-4b82-8df4-1a1ced93d9a3" (UID: "8316a979-44c7-4b82-8df4-1a1ced93d9a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:38.652498 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.652374    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-dshm" pod "8316a979-44c7-4b82-8df4-1a1ced93d9a3" (UID: "8316a979-44c7-4b82-8df4-1a1ced93d9a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:38.748345 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.748311    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:38.748345 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.748343    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgj4c\" (UniqueName: \"kubernetes.io/projected/8316a979-44c7-4b82-8df4-1a1ced93d9a3-kube-api-access-sgj4c\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:38.748345 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.748355    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8316a979-44c7-4b82-8df4-1a1ced93d9a3-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:38.748835 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:38.748366    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8316a979-44c7-4b82-8df4-1a1ced93d9a3-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:39.113517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:39.113358    5600 generic.go:356] "Generic (PLEG): container finished" podID="8316a979-44c7-4b82-8df4-1a1ced93d9a3" containerID="3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c" exitCode=0
Nov 03 20:12:39.113517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:39.113405    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j" event={"ID":"8316a979-44c7-4b82-8df4-1a1ced93d9a3","Type":"ContainerDied","Data":"3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c"}
Nov 03 20:12:39.113517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:39.113457    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j" event={"ID":"8316a979-44c7-4b82-8df4-1a1ced93d9a3","Type":"ContainerDied","Data":"21c497e3abae61ef1df170e65462ca61d8aa42d032cfb6546af8d4cca1638d7b"}
Nov 03 20:12:39.113517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:39.113482    5600 scope.go:122] "RemoveContainer" containerID="3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c"
Nov 03 20:12:39.113849 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:39.113643    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"
Nov 03 20:12:39.146368 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:39.146017    5600 scope.go:122] "RemoveContainer" containerID="3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c"
Nov 03 20:12:39.146487 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:12:39.146363    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c\": container with ID starting with 3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c not found: ID does not exist" containerID="3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c"
Nov 03 20:12:39.146487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:39.146403    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c"} err="failed to get container status \"3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c\": rpc error: code = NotFound desc = could not find container \"3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c\": container with ID starting with 3697163a5986324c53f25a3d0e616ae068bdc157d9a8738eeb700c1419f5b93c not found: ID does not exist"
Nov 03 20:12:39.165085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:39.165045    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"]
Nov 03 20:12:39.184486 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:39.178833    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-acd98d21/e2e-lifecycle-model-acd98d21-kserve-5964c7f9c7-xln8j"]
Nov 03 20:12:39.620569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:39.620532    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8316a979-44c7-4b82-8df4-1a1ced93d9a3" path="/var/lib/kubelet/pods/8316a979-44c7-4b82-8df4-1a1ced93d9a3/volumes"
Nov 03 20:12:40.862401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:40.862363    5600 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:12:40.862879 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:40.862418    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:12:43.463087 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.463022    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:12:43.497755 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.497710    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"]
Nov 03 20:12:43.499144 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.499105    5600 cpu_manager.go:414] "containerMap: removing container" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy"
Nov 03 20:12:43.499289 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.499272    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy"
Nov 03 20:12:43.499467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.499421    5600 cpu_manager.go:414] "containerMap: removing container" podUID="7c655eef-f597-485d-aa7c-543b615096c3" containerName="revoke-keys"
Nov 03 20:12:43.499588 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.499574    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c655eef-f597-485d-aa7c-543b615096c3" containerName="revoke-keys"
Nov 03 20:12:43.499822 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.499799    5600 cpu_manager.go:414] "containerMap: removing container" podUID="8316a979-44c7-4b82-8df4-1a1ced93d9a3" containerName="main"
Nov 03 20:12:43.499987 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.499936    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8316a979-44c7-4b82-8df4-1a1ced93d9a3" containerName="main"
Nov 03 20:12:43.500104 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.499843    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-data" pod "0ae75d0d-c70f-4206-9eeb-f02c9b723626" (UID: "0ae75d0d-c70f-4206-9eeb-f02c9b723626"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:43.500168 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.499578    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-data\") pod \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") "
Nov 03 20:12:43.500258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.500237    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istiod-ca-cert\") pod \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") "
Nov 03 20:12:43.500303 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.500288    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="8316a979-44c7-4b82-8df4-1a1ced93d9a3" containerName="main"
Nov 03 20:12:43.500347 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.500302    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-certs\") pod \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") "
Nov 03 20:12:43.500347 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.500321    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerName="istio-proxy"
Nov 03 20:12:43.500347 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.500333    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c655eef-f597-485d-aa7c-543b615096c3" containerName="revoke-keys"
Nov 03 20:12:43.500347 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.500341    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-envoy\") pod \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") "
Nov 03 20:12:43.500774 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.500413    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-token\" (UniqueName: \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-token\") pod \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") "
Nov 03 20:12:43.500774 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.500552    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-credential-socket\") pod \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") "
Nov 03 20:12:43.500774 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.500704    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-kube-api-access-ftpnz\" (UniqueName: \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-kube-api-access-ftpnz\") pod \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") "
Nov 03 20:12:43.500898 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.500786    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-socket\") pod \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") "
Nov 03 20:12:43.500944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.500929    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-podinfo\") pod \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\" (UID: \"0ae75d0d-c70f-4206-9eeb-f02c9b723626\") "
Nov 03 20:12:43.501710 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.501690    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.503135 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.502782    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-socket" pod "0ae75d0d-c70f-4206-9eeb-f02c9b723626" (UID: "0ae75d0d-c70f-4206-9eeb-f02c9b723626"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:43.503135 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.502814    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-certs" pod "0ae75d0d-c70f-4206-9eeb-f02c9b723626" (UID: "0ae75d0d-c70f-4206-9eeb-f02c9b723626"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:43.503135 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.502948    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-credential-socket" pod "0ae75d0d-c70f-4206-9eeb-f02c9b723626" (UID: "0ae75d0d-c70f-4206-9eeb-f02c9b723626"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:43.503135 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.503099    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istiod-ca-cert" pod "0ae75d0d-c70f-4206-9eeb-f02c9b723626" (UID: "0ae75d0d-c70f-4206-9eeb-f02c9b723626"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:43.509767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.508594    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.519807 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.519717    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-acd98d21-openshift-default-dockercfg-b8v8z\""
Nov 03 20:12:43.540899 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.540850    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"]
Nov 03 20:12:43.560381 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.559659    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-kube-api-access-ftpnz" pod "0ae75d0d-c70f-4206-9eeb-f02c9b723626" (UID: "0ae75d0d-c70f-4206-9eeb-f02c9b723626"). InnerVolumeSpecName "kube-api-access-ftpnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:43.560381 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.559801    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-podinfo" pod "0ae75d0d-c70f-4206-9eeb-f02c9b723626" (UID: "0ae75d0d-c70f-4206-9eeb-f02c9b723626"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:43.560942 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.560725    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-envoy" pod "0ae75d0d-c70f-4206-9eeb-f02c9b723626" (UID: "0ae75d0d-c70f-4206-9eeb-f02c9b723626"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:43.564833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.562212    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-token" pod "0ae75d0d-c70f-4206-9eeb-f02c9b723626" (UID: "0ae75d0d-c70f-4206-9eeb-f02c9b723626"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:43.603230 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.603186    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162731d0-9357-47d8-a643-f692c871a6b6-istio-podinfo\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.603403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.603318    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-certs\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.603515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.603425    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162731d0-9357-47d8-a643-f692c871a6b6-istiod-ca-cert\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.603515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.603494    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsd7\" (UniqueName: \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-kube-api-access-tjsd7\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.603642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.603570    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-credential-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.603642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.603601    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-envoy\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.603642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.603626    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-istio-token\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.603777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.603671    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-data\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.603848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.603828    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.604016 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.604000    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.605012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.604018    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.605012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.604030    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.605012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.604039    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.605012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.604047    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.605012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.604055    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.605012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.604064    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftpnz\" (UniqueName: \"kubernetes.io/projected/0ae75d0d-c70f-4206-9eeb-f02c9b723626-kube-api-access-ftpnz\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.605012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.604074    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae75d0d-c70f-4206-9eeb-f02c9b723626-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.708624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.708588    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162731d0-9357-47d8-a643-f692c871a6b6-istiod-ca-cert\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.708814 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.708637    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsd7\" (UniqueName: \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-kube-api-access-tjsd7\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.708814 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.708721    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-credential-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.708814 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.708748    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-envoy\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.708814 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.708771    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-istio-token\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.708814 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.708809    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-data\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.709107 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.708988    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.709107 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.709086    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162731d0-9357-47d8-a643-f692c871a6b6-istio-podinfo\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.710947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.709212    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-certs\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.710947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.709837    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-data\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.710947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.710115    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-credential-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.710947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.710219    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162731d0-9357-47d8-a643-f692c871a6b6-istiod-ca-cert\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.710947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.710427    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.711274 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.711248    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-certs\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.715269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.715214    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-envoy\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.720940 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.720889    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162731d0-9357-47d8-a643-f692c871a6b6-istio-podinfo\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.723272 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.723231    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-istio-token\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.725061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.725041    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsd7\" (UniqueName: \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-kube-api-access-tjsd7\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:43.776266 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.776240    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:12:43.810711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.810677    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2878c757-c831-4404-9182-8bbb34d3274b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2878c757-c831-4404-9182-8bbb34d3274b-istiod-ca-cert\") pod \"2878c757-c831-4404-9182-8bbb34d3274b\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") "
Nov 03 20:12:43.810900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.810754    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-socket\") pod \"2878c757-c831-4404-9182-8bbb34d3274b\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") "
Nov 03 20:12:43.810900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.810790    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-credential-socket\") pod \"2878c757-c831-4404-9182-8bbb34d3274b\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") "
Nov 03 20:12:43.810900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.810831    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2878c757-c831-4404-9182-8bbb34d3274b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2878c757-c831-4404-9182-8bbb34d3274b-istio-podinfo\") pod \"2878c757-c831-4404-9182-8bbb34d3274b\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") "
Nov 03 20:12:43.810900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.810860    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-istio-token\" (UniqueName: \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-istio-token\") pod \"2878c757-c831-4404-9182-8bbb34d3274b\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") "
Nov 03 20:12:43.811113 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.811009    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-kube-api-access-hlhcb\" (UniqueName: \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-kube-api-access-hlhcb\") pod \"2878c757-c831-4404-9182-8bbb34d3274b\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") "
Nov 03 20:12:43.811218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.811201    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-envoy\") pod \"2878c757-c831-4404-9182-8bbb34d3274b\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") "
Nov 03 20:12:43.811278 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.811235    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-certs\") pod \"2878c757-c831-4404-9182-8bbb34d3274b\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") "
Nov 03 20:12:43.811372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.811354    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-data\") pod \"2878c757-c831-4404-9182-8bbb34d3274b\" (UID: \"2878c757-c831-4404-9182-8bbb34d3274b\") "
Nov 03 20:12:43.812561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.812226    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-data" pod "2878c757-c831-4404-9182-8bbb34d3274b" (UID: "2878c757-c831-4404-9182-8bbb34d3274b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:43.813318 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.812676    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2878c757-c831-4404-9182-8bbb34d3274b-istiod-ca-cert" pod "2878c757-c831-4404-9182-8bbb34d3274b" (UID: "2878c757-c831-4404-9182-8bbb34d3274b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:43.813318 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.812941    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-socket" pod "2878c757-c831-4404-9182-8bbb34d3274b" (UID: "2878c757-c831-4404-9182-8bbb34d3274b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:43.813318 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.813113    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-credential-socket" pod "2878c757-c831-4404-9182-8bbb34d3274b" (UID: "2878c757-c831-4404-9182-8bbb34d3274b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:43.814692 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.814666    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-certs" pod "2878c757-c831-4404-9182-8bbb34d3274b" (UID: "2878c757-c831-4404-9182-8bbb34d3274b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:43.818086 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.818050    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2878c757-c831-4404-9182-8bbb34d3274b-istio-podinfo" pod "2878c757-c831-4404-9182-8bbb34d3274b" (UID: "2878c757-c831-4404-9182-8bbb34d3274b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:43.818239 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.818220    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-istio-token" pod "2878c757-c831-4404-9182-8bbb34d3274b" (UID: "2878c757-c831-4404-9182-8bbb34d3274b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:43.819753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.819730    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-envoy" pod "2878c757-c831-4404-9182-8bbb34d3274b" (UID: "2878c757-c831-4404-9182-8bbb34d3274b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:43.820076 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.820051    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-kube-api-access-hlhcb" pod "2878c757-c831-4404-9182-8bbb34d3274b" (UID: "2878c757-c831-4404-9182-8bbb34d3274b"). InnerVolumeSpecName "kube-api-access-hlhcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:43.914419 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.914378    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.914419 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.914419    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.914683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.914473    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.914683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.914490    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2878c757-c831-4404-9182-8bbb34d3274b-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.914683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.914505    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.914683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.914519    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2878c757-c831-4404-9182-8bbb34d3274b-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.914683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.914532    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2878c757-c831-4404-9182-8bbb34d3274b-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.914683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.914546    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.914683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.914559    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlhcb\" (UniqueName: \"kubernetes.io/projected/2878c757-c831-4404-9182-8bbb34d3274b-kube-api-access-hlhcb\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:43.926173 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:43.926125    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:44.157063 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.157024    5600 generic.go:356] "Generic (PLEG): container finished" podID="2878c757-c831-4404-9182-8bbb34d3274b" containerID="021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f" exitCode=0
Nov 03 20:12:44.157267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.157079    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"
Nov 03 20:12:44.157267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.157111    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" event={"ID":"2878c757-c831-4404-9182-8bbb34d3274b","Type":"ContainerDied","Data":"021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f"}
Nov 03 20:12:44.157267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.157178    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7" event={"ID":"2878c757-c831-4404-9182-8bbb34d3274b","Type":"ContainerDied","Data":"9d38c416eca11baa9d4c1a1f556213844b3c49276b452bb2a0f4b2cd05cb2464"}
Nov 03 20:12:44.157267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.157209    5600 scope.go:122] "RemoveContainer" containerID="021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f"
Nov 03 20:12:44.158672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.158644    5600 generic.go:356] "Generic (PLEG): container finished" podID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" containerID="00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263" exitCode=0
Nov 03 20:12:44.158769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.158684    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"
Nov 03 20:12:44.158769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.158711    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" event={"ID":"0ae75d0d-c70f-4206-9eeb-f02c9b723626","Type":"ContainerDied","Data":"00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263"}
Nov 03 20:12:44.158769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.158754    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc" event={"ID":"0ae75d0d-c70f-4206-9eeb-f02c9b723626","Type":"ContainerDied","Data":"8dbca043c8dd232a0bd95ac5062dd34325614d8dee704dfee013b9cff080bf2e"}
Nov 03 20:12:44.169305 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.169286    5600 scope.go:122] "RemoveContainer" containerID="021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f"
Nov 03 20:12:44.169591 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:12:44.169573    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f\": container with ID starting with 021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f not found: ID does not exist" containerID="021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f"
Nov 03 20:12:44.169658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.169601    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f"} err="failed to get container status \"021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f\": rpc error: code = NotFound desc = could not find container \"021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f\": container with ID starting with 021d82f47cde01948e4e6aa3c8242fd2024bb9e06b514b8d6cd99828721dc88f not found: ID does not exist"
Nov 03 20:12:44.169658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.169617    5600 scope.go:122] "RemoveContainer" containerID="00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263"
Nov 03 20:12:44.179066 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.179050    5600 scope.go:122] "RemoveContainer" containerID="00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263"
Nov 03 20:12:44.179334 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:12:44.179317    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263\": container with ID starting with 00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263 not found: ID does not exist" containerID="00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263"
Nov 03 20:12:44.179384 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.179344    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263"} err="failed to get container status \"00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263\": rpc error: code = NotFound desc = could not find container \"00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263\": container with ID starting with 00074aa5361382c0ed020ad131ddcab5702dce014d944ab7bb1458825e82d263 not found: ID does not exist"
Nov 03 20:12:44.204912 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.204880    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"]
Nov 03 20:12:44.221832 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.221801    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-44qkc"]
Nov 03 20:12:44.248206 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.248177    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"]
Nov 03 20:12:44.265301 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.265232    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-dbgl7"]
Nov 03 20:12:44.292273 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.292239    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"]
Nov 03 20:12:44.300352 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:12:44.300324    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162731d0_9357_47d8_a643_f692c871a6b6.slice/crio-73ebe9ed812716126585362ee4e89e0e17b3506f9445c72e4ee0329856e9d189 WatchSource:0}: Error finding container 73ebe9ed812716126585362ee4e89e0e17b3506f9445c72e4ee0329856e9d189: Status 404 returned error can't find the container with id 73ebe9ed812716126585362ee4e89e0e17b3506f9445c72e4ee0329856e9d189
Nov 03 20:12:44.303757 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.303710    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:44.303873 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.303792    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:44.303873 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:44.303825    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:12:45.172100 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:45.172062    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" event={"ID":"162731d0-9357-47d8-a643-f692c871a6b6","Type":"ContainerStarted","Data":"868a15a0ffbe579544238fd90ec2086b58ce073fde50894a7bee15fdac42b0b6"}
Nov 03 20:12:45.172701 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:45.172122    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" event={"ID":"162731d0-9357-47d8-a643-f692c871a6b6","Type":"ContainerStarted","Data":"73ebe9ed812716126585362ee4e89e0e17b3506f9445c72e4ee0329856e9d189"}
Nov 03 20:12:45.218503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:45.217204    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" podStartSLOduration=16.217169162 podStartE2EDuration="16.217169162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:45.214803582 +0000 UTC m=+1947.727625410" watchObservedRunningTime="2026-11-03 20:12:45.217169162 +0000 UTC m=+1947.729990957"
Nov 03 20:12:45.626491 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:45.626397    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae75d0d-c70f-4206-9eeb-f02c9b723626" path="/var/lib/kubelet/pods/0ae75d0d-c70f-4206-9eeb-f02c9b723626/volumes"
Nov 03 20:12:45.627123 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:45.627105    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2878c757-c831-4404-9182-8bbb34d3274b" path="/var/lib/kubelet/pods/2878c757-c831-4404-9182-8bbb34d3274b/volumes"
Nov 03 20:12:45.809473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:45.808973    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:12:45.931761 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:45.931717    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:45.933355 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:45.933320    5600 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:12:45.933570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:45.933368    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" podUID="162731d0-9357-47d8-a643-f692c871a6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:12:46.926548 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:46.926514    5600 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:12:46.927145 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:46.927103    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" podUID="162731d0-9357-47d8-a643-f692c871a6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:12:47.943910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:47.943869    5600 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:12:47.944569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:47.944531    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" podUID="162731d0-9357-47d8-a643-f692c871a6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:12:48.129130 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:48.129093    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"]
Nov 03 20:12:48.166795 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:48.166741    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b918c5d-zxs76"]
Nov 03 20:12:48.926898 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:48.926859    5600 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:12:48.927098 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:48.926914    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" podUID="162731d0-9357-47d8-a643-f692c871a6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:12:49.651221 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:49.651183    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c655eef-f597-485d-aa7c-543b615096c3" path="/var/lib/kubelet/pods/7c655eef-f597-485d-aa7c-543b615096c3/volumes"
Nov 03 20:12:49.927095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:49.926764    5600 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:12:49.927095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:49.926827    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" podUID="162731d0-9357-47d8-a643-f692c871a6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:12:50.882199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.882164    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"]
Nov 03 20:12:50.883322 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.883299    5600 cpu_manager.go:414] "containerMap: removing container" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy"
Nov 03 20:12:50.883366 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.883327    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy"
Nov 03 20:12:50.883533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.883520    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="2878c757-c831-4404-9182-8bbb34d3274b" containerName="istio-proxy"
Nov 03 20:12:50.894186 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.892395    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:50.911714 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.905820    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"]
Nov 03 20:12:50.911714 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.910630    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-663013-serving-cert\""
Nov 03 20:12:50.936130 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.928199    5600 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:12:50.936130 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.928327    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" podUID="162731d0-9357-47d8-a643-f692c871a6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:12:50.990458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.988687    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfppt\" (UniqueName: \"kubernetes.io/projected/34d8f77d-4a5f-479e-8c60-32d52a23b253-kube-api-access-zfppt\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:50.990458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.989174    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-maas-api-tls\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:50.990458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.989361    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-metrics-certs\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:50.990458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:50.989824    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d8f77d-4a5f-479e-8c60-32d52a23b253-openshift-service-ca\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:51.120525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:51.106514    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d8f77d-4a5f-479e-8c60-32d52a23b253-openshift-service-ca\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:51.120904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:51.107426    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d8f77d-4a5f-479e-8c60-32d52a23b253-openshift-service-ca\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:51.126887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:51.122631    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfppt\" (UniqueName: \"kubernetes.io/projected/34d8f77d-4a5f-479e-8c60-32d52a23b253-kube-api-access-zfppt\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:51.126887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:51.122830    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-maas-api-tls\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:51.126887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:51.122914    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-metrics-certs\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:51.137887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:51.131605    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-maas-api-tls\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:51.137887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:51.133168    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-metrics-certs\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:51.159613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:51.158903    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfppt\" (UniqueName: \"kubernetes.io/projected/34d8f77d-4a5f-479e-8c60-32d52a23b253-kube-api-access-zfppt\") pod \"maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:51.213157 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:51.213120    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:52.143804 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:52.135561    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"]
Nov 03 20:12:52.252549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:52.252511    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:52.253729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:52.253584    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:52.255488 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:52.254720    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:12:52.459167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:52.458918    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs" event={"ID":"34d8f77d-4a5f-479e-8c60-32d52a23b253","Type":"ContainerStarted","Data":"c7895b17a8a8d91266ea4dab4a1b753366ade28787510cac70bd8daf06c02d5a"}
Nov 03 20:12:53.469816 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:53.469775    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs" event={"ID":"34d8f77d-4a5f-479e-8c60-32d52a23b253","Type":"ContainerStarted","Data":"c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9"}
Nov 03 20:12:53.470350 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:53.469841    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:12:53.492837 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:53.492780    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs" podStartSLOduration=2.983127442 podStartE2EDuration="3.492763653s" totalImagesPullingTime="509.636211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:53.488272739 +0000 UTC m=+1956.001094522" watchObservedRunningTime="2026-11-03 20:12:53.492763653 +0000 UTC m=+1956.005585474"
Nov 03 20:12:58.809010 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:12:58.808962    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:12:59.542761 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:12:59.542732    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:13:08.025462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:08.012883    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"]
Nov 03 20:13:08.026729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:08.026687    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-433102-bgzzk"]
Nov 03 20:13:08.457016 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:08.455809    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"]
Nov 03 20:13:08.457596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:08.457530    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" podUID="162731d0-9357-47d8-a643-f692c871a6b6" containerName="istio-proxy" containerID="cri-o://868a15a0ffbe579544238fd90ec2086b58ce073fde50894a7bee15fdac42b0b6" gracePeriod=30
Nov 03 20:13:09.626491 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:09.626394    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30019952-0426-409a-9b21-c42086a46970" path="/var/lib/kubelet/pods/30019952-0426-409a-9b21-c42086a46970/volumes"
Nov 03 20:13:13.936971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:13.936931    5600 generic.go:356] "Generic (PLEG): container finished" podID="162731d0-9357-47d8-a643-f692c871a6b6" containerID="868a15a0ffbe579544238fd90ec2086b58ce073fde50894a7bee15fdac42b0b6" exitCode=0
Nov 03 20:13:13.937833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:13.937627    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" event={"ID":"162731d0-9357-47d8-a643-f692c871a6b6","Type":"ContainerDied","Data":"868a15a0ffbe579544238fd90ec2086b58ce073fde50894a7bee15fdac42b0b6"}
Nov 03 20:13:14.460007 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.459979    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:13:14.608117 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.608080    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-envoy\") pod \"162731d0-9357-47d8-a643-f692c871a6b6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") "
Nov 03 20:13:14.608394 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.608373    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162731d0-9357-47d8-a643-f692c871a6b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162731d0-9357-47d8-a643-f692c871a6b6-istiod-ca-cert\") pod \"162731d0-9357-47d8-a643-f692c871a6b6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") "
Nov 03 20:13:14.608567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.608551    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-data\") pod \"162731d0-9357-47d8-a643-f692c871a6b6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") "
Nov 03 20:13:14.608687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.608673    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/162731d0-9357-47d8-a643-f692c871a6b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162731d0-9357-47d8-a643-f692c871a6b6-istio-podinfo\") pod \"162731d0-9357-47d8-a643-f692c871a6b6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") "
Nov 03 20:13:14.608917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.608901    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-kube-api-access-tjsd7\" (UniqueName: \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-kube-api-access-tjsd7\") pod \"162731d0-9357-47d8-a643-f692c871a6b6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") "
Nov 03 20:13:14.609121 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.609107    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-credential-socket\") pod \"162731d0-9357-47d8-a643-f692c871a6b6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") "
Nov 03 20:13:14.609364 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.609346    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-socket\") pod \"162731d0-9357-47d8-a643-f692c871a6b6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") "
Nov 03 20:13:14.609789 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.609769    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-certs\") pod \"162731d0-9357-47d8-a643-f692c871a6b6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") "
Nov 03 20:13:14.609938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.609924    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-istio-token\" (UniqueName: \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-istio-token\") pod \"162731d0-9357-47d8-a643-f692c871a6b6\" (UID: \"162731d0-9357-47d8-a643-f692c871a6b6\") "
Nov 03 20:13:14.699712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.699493    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-data" pod "162731d0-9357-47d8-a643-f692c871a6b6" (UID: "162731d0-9357-47d8-a643-f692c871a6b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:14.700570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.700107    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162731d0-9357-47d8-a643-f692c871a6b6-istiod-ca-cert" pod "162731d0-9357-47d8-a643-f692c871a6b6" (UID: "162731d0-9357-47d8-a643-f692c871a6b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:14.706958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.706928    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-credential-socket" pod "162731d0-9357-47d8-a643-f692c871a6b6" (UID: "162731d0-9357-47d8-a643-f692c871a6b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:14.707315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.707294    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-socket" pod "162731d0-9357-47d8-a643-f692c871a6b6" (UID: "162731d0-9357-47d8-a643-f692c871a6b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:14.707634 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.707616    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-certs" pod "162731d0-9357-47d8-a643-f692c871a6b6" (UID: "162731d0-9357-47d8-a643-f692c871a6b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:14.707850 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.707834    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-istio-token" pod "162731d0-9357-47d8-a643-f692c871a6b6" (UID: "162731d0-9357-47d8-a643-f692c871a6b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:14.721461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.716153    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:14.721461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.716187    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:14.721461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.716202    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:14.721461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.716216    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:14.721461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.716230    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162731d0-9357-47d8-a643-f692c871a6b6-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:14.721461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.716243    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:14.745152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.745097    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/162731d0-9357-47d8-a643-f692c871a6b6-istio-podinfo" pod "162731d0-9357-47d8-a643-f692c871a6b6" (UID: "162731d0-9357-47d8-a643-f692c871a6b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:13:14.853483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:14.831227    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162731d0-9357-47d8-a643-f692c871a6b6-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:15.007539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:15.007413    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-envoy" pod "162731d0-9357-47d8-a643-f692c871a6b6" (UID: "162731d0-9357-47d8-a643-f692c871a6b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:15.030476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:15.029995    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6" event={"ID":"162731d0-9357-47d8-a643-f692c871a6b6","Type":"ContainerDied","Data":"73ebe9ed812716126585362ee4e89e0e17b3506f9445c72e4ee0329856e9d189"}
Nov 03 20:13:15.030476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:15.030068    5600 scope.go:122] "RemoveContainer" containerID="868a15a0ffbe579544238fd90ec2086b58ce073fde50894a7bee15fdac42b0b6"
Nov 03 20:13:15.030476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:15.030236    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"
Nov 03 20:13:15.082583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:15.064738    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162731d0-9357-47d8-a643-f692c871a6b6-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:15.089736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:15.089641    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-kube-api-access-tjsd7" pod "162731d0-9357-47d8-a643-f692c871a6b6" (UID: "162731d0-9357-47d8-a643-f692c871a6b6"). InnerVolumeSpecName "kube-api-access-tjsd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:15.210928 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:15.210883    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjsd7\" (UniqueName: \"kubernetes.io/projected/162731d0-9357-47d8-a643-f692c871a6b6-kube-api-access-tjsd7\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:15.617573 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:15.617515    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"]
Nov 03 20:13:15.672470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:15.668381    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:13:15.803457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:15.803004    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:15.839310 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:15.837573    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-vlxq6"]
Nov 03 20:13:16.333462 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:13:16.315345    5600 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162731d0_9357_47d8_a643_f692c871a6b6.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 20:13:16.628946 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:16.624850    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:16.695461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:16.688529    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:13:16.695461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:16.689983    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:13:16.806461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:16.805527    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:13:16.810459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:16.806999    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:13:16.945460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:16.944929    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:13:17.574307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:17.574262    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:17.656495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:17.630626    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:17.696569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:17.696522    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:17.750781 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:17.750277    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:13:18.006467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:17.998317    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162731d0-9357-47d8-a643-f692c871a6b6" path="/var/lib/kubelet/pods/162731d0-9357-47d8-a643-f692c871a6b6/volumes"
Nov 03 20:13:18.008567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:18.008543    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:13:18.476716 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:18.474543    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 03 20:13:18.620028 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:18.619587    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:13:18.648584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:18.648550    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:13:18.651614 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:18.651583    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:13:18.655584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:18.655543    5600 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:18.747288 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:18.746556    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:18.859461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:18.764910    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:13:18.863305 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:18.861337    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:13:18.863305 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:18.822611    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:13:19.343644 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:19.337829    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:13:19.474829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:19.474797    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:13:19.627243 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:19.627142    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:20.020826 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:19.749114    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:13:20.662783 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:20.662721    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:20.954699 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:20.954657    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:21.029591 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.029542    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:21.059563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.059520    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:21.269343 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.266852    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:13:21.342914 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.342868    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:13:21.343166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.343148    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"]
Nov 03 20:13:21.344672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.344603    5600 cpu_manager.go:414] "containerMap: removing container" podUID="162731d0-9357-47d8-a643-f692c871a6b6" containerName="istio-proxy"
Nov 03 20:13:21.345778 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.345760    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162731d0-9357-47d8-a643-f692c871a6b6" containerName="istio-proxy"
Nov 03 20:13:21.346091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.346073    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="162731d0-9357-47d8-a643-f692c871a6b6" containerName="istio-proxy"
Nov 03 20:13:21.374957 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.353853    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.455460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.374772    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"]
Nov 03 20:13:21.455460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.392958    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2092ba-serving-cert\""
Nov 03 20:13:21.455460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.430245    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csnll\" (UniqueName: \"kubernetes.io/projected/ebac9276-2fb6-4924-9b00-5e5e589ba683-kube-api-access-csnll\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.455460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.430372    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-maas-api-tls\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.455460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.430423    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebac9276-2fb6-4924-9b00-5e5e589ba683-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.455460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.430582    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-metrics-certs\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.591526 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.567551    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csnll\" (UniqueName: \"kubernetes.io/projected/ebac9276-2fb6-4924-9b00-5e5e589ba683-kube-api-access-csnll\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.677063 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.677023    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-maas-api-tls\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.681458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.679568    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebac9276-2fb6-4924-9b00-5e5e589ba683-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.681458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.679707    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-metrics-certs\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.681458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.665476    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:13:21.746980 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.746398    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-maas-api-tls\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.814363 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.814326    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csnll\" (UniqueName: \"kubernetes.io/projected/ebac9276-2fb6-4924-9b00-5e5e589ba683-kube-api-access-csnll\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.915822 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.915071    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:21.974465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.970528    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:21.978119 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.978047    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebac9276-2fb6-4924-9b00-5e5e589ba683-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:21.993912 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:21.993878    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:13:22.192507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:22.170377    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-metrics-certs\") pod \"maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:22.374581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:22.370253    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:22.724581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:22.724534    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:22.817304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:22.817259    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:23.007505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:22.989113    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:13:23.008957 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:22.994662    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:23.013454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:23.010223    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:13:23.214124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:23.206481    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:13:23.554548 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:23.553285    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:13:23.605781 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:23.605741    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"]
Nov 03 20:13:23.878503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:23.877308    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:13:24.029458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:24.020088    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"]
Nov 03 20:13:24.029458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:24.020338    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerName="istio-proxy" containerID="cri-o://8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd" gracePeriod=30
Nov 03 20:13:24.497269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:24.497217    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d" event={"ID":"ebac9276-2fb6-4924-9b00-5e5e589ba683","Type":"ContainerStarted","Data":"a94fdb2ec555239c2aa4b202c951793d04d4a58ec6af0de59c7857a6cc1eb242"}
Nov 03 20:13:25.531300 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:25.524718    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d" event={"ID":"ebac9276-2fb6-4924-9b00-5e5e589ba683","Type":"ContainerStarted","Data":"4033461df0e38a887361737786f5cd6a6450aff29491222d5bde00a1a4629e2a"}
Nov 03 20:13:25.531300 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:25.525948    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:25.549988 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:25.549924    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d" podStartSLOduration=3.865626658 podStartE2EDuration="4.549904917s" totalImagesPullingTime="684.278259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:25.546361223 +0000 UTC m=+1988.059183005" watchObservedRunningTime="2026-11-03 20:13:25.549904917 +0000 UTC m=+1988.062726702"
Nov 03 20:13:27.408383 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.404947    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"]
Nov 03 20:13:27.410806 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.410780    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.417496 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.417113    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f817b0\"/\"openshift-service-ca.crt\""
Nov 03 20:13:27.417496 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.417229    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f817b0\"/\"kube-root-ca.crt\""
Nov 03 20:13:27.417496 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.417344    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f817b0\"/\"default-dockercfg-wkxr8\""
Nov 03 20:13:27.417496 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.417423    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f817b0\"/\"test-model-f817b0-kserve-self-signed-certs\""
Nov 03 20:13:27.425650 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.425601    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"]
Nov 03 20:13:27.559489 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.559367    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-tmp-dir\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.559732 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.559642    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-model-cache\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.559732 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.559711    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8r9j\" (UniqueName: \"kubernetes.io/projected/f9a8989b-01b5-4041-b0b4-63591932d918-kube-api-access-s8r9j\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.559869 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.559796    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a8989b-01b5-4041-b0b4-63591932d918-tls-certs\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.559969 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.559948    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-home\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.560039 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.560022    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-dshm\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.673706 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.673615    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-model-cache\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.673881 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.673708    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8r9j\" (UniqueName: \"kubernetes.io/projected/f9a8989b-01b5-4041-b0b4-63591932d918-kube-api-access-s8r9j\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.673881 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.673738    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a8989b-01b5-4041-b0b4-63591932d918-tls-certs\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.673881 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.673862    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-home\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.674154 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.673937    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-dshm\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.674154 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.674045    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-tmp-dir\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.674598 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.674571    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-tmp-dir\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.674866 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.674843    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-model-cache\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.689456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.686355    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-home\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.702557 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.702077    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8r9j\" (UniqueName: \"kubernetes.io/projected/f9a8989b-01b5-4041-b0b4-63591932d918-kube-api-access-s8r9j\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.702557 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.702368    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-dshm\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.755105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.753262    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a8989b-01b5-4041-b0b4-63591932d918-tls-certs\") pod \"test-model-f817b0-kserve-8479d856d7-x4vpr\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") " pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:27.755105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:27.754885    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:28.207909 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:13:28.207860    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9a8989b_01b5_4041_b0b4_63591932d918.slice/crio-1aae8c26326d8ec4a8c6168d380e1ff7c28bf5d6c2786453d1925b40a93b6216 WatchSource:0}: Error finding container 1aae8c26326d8ec4a8c6168d380e1ff7c28bf5d6c2786453d1925b40a93b6216: Status 404 returned error can't find the container with id 1aae8c26326d8ec4a8c6168d380e1ff7c28bf5d6c2786453d1925b40a93b6216
Nov 03 20:13:28.218910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:28.213906    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"]
Nov 03 20:13:28.594999 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:28.594908    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr" event={"ID":"f9a8989b-01b5-4041-b0b4-63591932d918","Type":"ContainerStarted","Data":"4494046154f63eb743c439be9d1f52d438b0517251b42089421277a006a536ee"}
Nov 03 20:13:28.594999 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:28.594955    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr" event={"ID":"f9a8989b-01b5-4041-b0b4-63591932d918","Type":"ContainerStarted","Data":"1aae8c26326d8ec4a8c6168d380e1ff7c28bf5d6c2786453d1925b40a93b6216"}
Nov 03 20:13:28.619502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:28.619417    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr" podStartSLOduration=1.6193947020000001 podStartE2EDuration="1.619394702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:28.613533472 +0000 UTC m=+1991.126355261" watchObservedRunningTime="2026-11-03 20:13:28.619394702 +0000 UTC m=+1991.132216485"
Nov 03 20:13:28.760463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:28.760372    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:28.815050 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:13:28.814969    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:13:29.422870 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.422834    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:13:29.468226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.466703    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e39420c-d71a-475a-8f31-42ab6a7ea412-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e39420c-d71a-475a-8f31-42ab6a7ea412-istiod-ca-cert\") pod \"0e39420c-d71a-475a-8f31-42ab6a7ea412\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") "
Nov 03 20:13:29.468226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.466755    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-credential-socket\") pod \"0e39420c-d71a-475a-8f31-42ab6a7ea412\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") "
Nov 03 20:13:29.468226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.466797    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-podinfo\") pod \"0e39420c-d71a-475a-8f31-42ab6a7ea412\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") "
Nov 03 20:13:29.468226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.466902    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-certs\") pod \"0e39420c-d71a-475a-8f31-42ab6a7ea412\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") "
Nov 03 20:13:29.468226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.466954    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-socket\") pod \"0e39420c-d71a-475a-8f31-42ab6a7ea412\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") "
Nov 03 20:13:29.468226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.467059    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-data\") pod \"0e39420c-d71a-475a-8f31-42ab6a7ea412\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") "
Nov 03 20:13:29.468226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.467274    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-kube-api-access-stl5x\" (UniqueName: \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-kube-api-access-stl5x\") pod \"0e39420c-d71a-475a-8f31-42ab6a7ea412\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") "
Nov 03 20:13:29.468226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.467402    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-envoy\") pod \"0e39420c-d71a-475a-8f31-42ab6a7ea412\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") "
Nov 03 20:13:29.468226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.467458    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-token\" (UniqueName: \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-token\") pod \"0e39420c-d71a-475a-8f31-42ab6a7ea412\" (UID: \"0e39420c-d71a-475a-8f31-42ab6a7ea412\") "
Nov 03 20:13:29.468226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.467884    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-certs" pod "0e39420c-d71a-475a-8f31-42ab6a7ea412" (UID: "0e39420c-d71a-475a-8f31-42ab6a7ea412"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:29.469347 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.468286    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:29.469347 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.468625    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e39420c-d71a-475a-8f31-42ab6a7ea412-istiod-ca-cert" pod "0e39420c-d71a-475a-8f31-42ab6a7ea412" (UID: "0e39420c-d71a-475a-8f31-42ab6a7ea412"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:29.469347 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.468665    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-credential-socket" pod "0e39420c-d71a-475a-8f31-42ab6a7ea412" (UID: "0e39420c-d71a-475a-8f31-42ab6a7ea412"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:29.469838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.469659    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-data" pod "0e39420c-d71a-475a-8f31-42ab6a7ea412" (UID: "0e39420c-d71a-475a-8f31-42ab6a7ea412"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:29.469838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.469724    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-socket" pod "0e39420c-d71a-475a-8f31-42ab6a7ea412" (UID: "0e39420c-d71a-475a-8f31-42ab6a7ea412"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:29.514140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.513665    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-podinfo" pod "0e39420c-d71a-475a-8f31-42ab6a7ea412" (UID: "0e39420c-d71a-475a-8f31-42ab6a7ea412"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:13:29.518958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.518906    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-token" pod "0e39420c-d71a-475a-8f31-42ab6a7ea412" (UID: "0e39420c-d71a-475a-8f31-42ab6a7ea412"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:29.519101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.519039    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-envoy" pod "0e39420c-d71a-475a-8f31-42ab6a7ea412" (UID: "0e39420c-d71a-475a-8f31-42ab6a7ea412"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:29.519641 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.519605    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-kube-api-access-stl5x" pod "0e39420c-d71a-475a-8f31-42ab6a7ea412" (UID: "0e39420c-d71a-475a-8f31-42ab6a7ea412"). InnerVolumeSpecName "kube-api-access-stl5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:29.569995 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.569819    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:29.570196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.570001    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:29.570196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.570044    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:29.570196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.570061    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:29.570196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.570077    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stl5x\" (UniqueName: \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-kube-api-access-stl5x\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:29.570196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.570094    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:29.570196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.570109    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e39420c-d71a-475a-8f31-42ab6a7ea412-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:29.570196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.570124    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e39420c-d71a-475a-8f31-42ab6a7ea412-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:29.613332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.613204    5600 generic.go:356] "Generic (PLEG): container finished" podID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerID="8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd" exitCode=0
Nov 03 20:13:29.615505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.613906    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"
Nov 03 20:13:29.626319 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.626279    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" event={"ID":"0e39420c-d71a-475a-8f31-42ab6a7ea412","Type":"ContainerDied","Data":"8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd"}
Nov 03 20:13:29.626513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.626339    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9" event={"ID":"0e39420c-d71a-475a-8f31-42ab6a7ea412","Type":"ContainerDied","Data":"5f99eff7cf2968361297c258f566afdd17d294988087f91d6a3da6b89ace820f"}
Nov 03 20:13:29.626513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.626373    5600 scope.go:122] "RemoveContainer" containerID="8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd"
Nov 03 20:13:29.646538 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.646500    5600 scope.go:122] "RemoveContainer" containerID="8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd"
Nov 03 20:13:29.646921 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:13:29.646900    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd\": container with ID starting with 8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd not found: ID does not exist" containerID="8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd"
Nov 03 20:13:29.646998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.646934    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd"} err="failed to get container status \"8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd\": rpc error: code = NotFound desc = could not find container \"8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd\": container with ID starting with 8f9657ea946149e9dc5c43334623c6b97e55d2c577a52e0c50f1e2b058345fcd not found: ID does not exist"
Nov 03 20:13:29.655230 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.655202    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"]
Nov 03 20:13:29.660147 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:29.660124    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-5mgv9"]
Nov 03 20:13:31.619465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:31.619361    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" path="/var/lib/kubelet/pods/0e39420c-d71a-475a-8f31-42ab6a7ea412/volumes"
Nov 03 20:13:31.643073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:31.643033    5600 generic.go:356] "Generic (PLEG): container finished" podID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerID="1e37cbfa6f62d5c14de64e1b710182d24fa494a379f0632b2389f04fb5905a15" exitCode=0
Nov 03 20:13:31.643254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:31.643105    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" event={"ID":"aec08d5e-0c0e-459e-85b4-0b80b5b60b35","Type":"ContainerDied","Data":"1e37cbfa6f62d5c14de64e1b710182d24fa494a379f0632b2389f04fb5905a15"}
Nov 03 20:13:31.643598 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:31.643583    5600 scope.go:122] "RemoveContainer" containerID="1e37cbfa6f62d5c14de64e1b710182d24fa494a379f0632b2389f04fb5905a15"
Nov 03 20:13:31.645717 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:31.645685    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:13:31.645833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:31.645778    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:13:31.645833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:31.645828    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:13:31.715636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:31.715599    5600 patch_prober.go:29] interesting pod/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:13:31.715895 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:31.715864    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:32.610454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:32.610285    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:13:32.653134 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:32.653101    5600 generic.go:356] "Generic (PLEG): container finished" podID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerID="cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830" exitCode=0
Nov 03 20:13:32.653624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:32.653172    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" event={"ID":"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec","Type":"ContainerDied","Data":"cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830"}
Nov 03 20:13:32.653719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:32.653699    5600 scope.go:122] "RemoveContainer" containerID="cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830"
Nov 03 20:13:32.655858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:32.655833    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" event={"ID":"aec08d5e-0c0e-459e-85b4-0b80b5b60b35","Type":"ContainerStarted","Data":"c06ae4f0a3d2359b78acee69204dfd186b60138111f2a8c6132f8801ea7f9dba"}
Nov 03 20:13:32.656061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:32.656031    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:13:32.656201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:32.656115    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:13:32.656201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:32.656160    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:13:32.749844 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:32.748892    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:13:32.750581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:32.750553    5600 patch_prober.go:29] interesting pod/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:13:32.750704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:32.750607    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 20:13:33.665906 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:33.665866    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" event={"ID":"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec","Type":"ContainerStarted","Data":"9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827"}
Nov 03 20:13:33.749488 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:33.749453    5600 patch_prober.go:29] interesting pod/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:13:33.749692 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:33.749510    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 20:13:34.354463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:34.347960    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:13:34.364506 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:34.359612    5600 patch_prober.go:29] interesting pod/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:13:34.364506 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:34.359676    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:13:34.749333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:34.749290    5600 patch_prober.go:29] interesting pod/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:13:34.749875 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:34.749353    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 20:13:35.321469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:35.320886    5600 patch_prober.go:29] interesting pod/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:13:35.321469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:35.320951    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:13:35.502460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:35.499123    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:13:35.775463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:35.771682    5600 patch_prober.go:29] interesting pod/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:13:35.775463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:35.771738    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 20:13:36.324463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:36.322846    5600 patch_prober.go:29] interesting pod/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:13:36.324463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:36.322925    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:13:36.751999 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:36.751927    5600 patch_prober.go:29] interesting pod/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:13:36.752221 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:36.751996    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 20:13:37.323880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:37.323834    5600 patch_prober.go:29] interesting pod/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:13:37.324399 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:37.323902    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:13:37.751202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:37.750785    5600 patch_prober.go:29] interesting pod/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:13:37.751202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:37.750843    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 20:13:37.761606 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:37.760517    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:37.774939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:37.774913    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:38.320481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:38.320424    5600 patch_prober.go:29] interesting pod/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:13:38.320763 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:38.320737    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:13:38.754949 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:38.754922    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:13:38.756060 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:38.756039    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:13:38.762858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:38.762839    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:13:38.959940 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:38.959905    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"]
Nov 03 20:13:38.961300 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:38.961247    5600 cpu_manager.go:414] "containerMap: removing container" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerName="istio-proxy"
Nov 03 20:13:38.961507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:38.961489    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerName="istio-proxy"
Nov 03 20:13:38.961870 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:38.961851    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e39420c-d71a-475a-8f31-42ab6a7ea412" containerName="istio-proxy"
Nov 03 20:13:38.966406 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:38.966386    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:38.972872 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:38.972848    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-babea4-serving-cert\""
Nov 03 20:13:39.074756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.074667    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-maas-api-tls\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.090416 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.090382    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686h6\" (UniqueName: \"kubernetes.io/projected/d21b7271-831b-42e4-8494-68277b1d0e10-kube-api-access-686h6\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.090688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.090668    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21b7271-831b-42e4-8494-68277b1d0e10-openshift-service-ca\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.118459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.094905    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-metrics-certs\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.208703 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.208665    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"]
Nov 03 20:13:39.234613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.233971    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-metrics-certs\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.234613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.234205    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-maas-api-tls\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.234613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.234281    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686h6\" (UniqueName: \"kubernetes.io/projected/d21b7271-831b-42e4-8494-68277b1d0e10-kube-api-access-686h6\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.234613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.234317    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21b7271-831b-42e4-8494-68277b1d0e10-openshift-service-ca\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.235365 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.235334    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21b7271-831b-42e4-8494-68277b1d0e10-openshift-service-ca\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.245112 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:13:39.245084    5600 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-serving-cert: secret "maas-api-e2e-inf-b-babea4-serving-cert" not found
Nov 03 20:13:39.350696 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:13:39.350607    5600 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-maas-api-tls podName:d21b7271-831b-42e4-8494-68277b1d0e10 nodeName:}" failed. No retries permitted until 2026-11-03 20:13:39.850564363 +0000 UTC m=+2002.363386145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-maas-api-tls") pod "maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x" (UID: "d21b7271-831b-42e4-8494-68277b1d0e10") : secret "maas-api-e2e-inf-b-babea4-serving-cert" not found
Nov 03 20:13:39.356458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.352401    5600 patch_prober.go:29] interesting pod/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:13:39.360557 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.360510    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:13:39.454944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.454898    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-metrics-certs\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.480565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.477470    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686h6\" (UniqueName: \"kubernetes.io/projected/d21b7271-831b-42e4-8494-68277b1d0e10-kube-api-access-686h6\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.918053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.918006    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-maas-api-tls\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:39.920810 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:39.920785    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-maas-api-tls\") pod \"maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:40.044977 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:40.044930    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:40.366476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:40.359546    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:13:40.380978 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:40.379168    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:13:40.382420 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:40.382390    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:13:40.512969 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:40.509642    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"]
Nov 03 20:13:40.525196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:40.524321    5600 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:13:40.769530 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:40.769445    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x" event={"ID":"d21b7271-831b-42e4-8494-68277b1d0e10","Type":"ContainerStarted","Data":"be7d6c76c3f0a5aa798ca17bcbec0d1da45d30b5c48c8798d45701c6d8bd6ad1"}
Nov 03 20:13:41.853632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:41.853574    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x" event={"ID":"d21b7271-831b-42e4-8494-68277b1d0e10","Type":"ContainerStarted","Data":"9682ebfbb505f6d4ddec5639a726d99c06db90af8d5b7ff2f7e6f43c51041b6c"}
Nov 03 20:13:41.854309 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:41.853752    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:41.891249 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:41.891172    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x" podStartSLOduration=3.379508317 podStartE2EDuration="3.891152413s" totalImagesPullingTime="511.644096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:41.888836191 +0000 UTC m=+2004.401657973" watchObservedRunningTime="2026-11-03 20:13:41.891152413 +0000 UTC m=+2004.403974196"
Nov 03 20:13:47.865088 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:13:47.865053    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:13:58.808918 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:13:58.808839    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:14:11.419463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:11.414992    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-28w6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:14:11.419463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:11.415095    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:24.075163 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.075129    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"]
Nov 03 20:14:24.082127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.082090    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.093138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.093111    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-babea4\"/\"kube-root-ca.crt\""
Nov 03 20:14:24.093371 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.093111    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-babea4\"/\"openshift-service-ca.crt\""
Nov 03 20:14:24.095979 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.095952    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"]
Nov 03 20:14:24.096779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.096760    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-babea4\"/\"default-dockercfg-hrdwg\""
Nov 03 20:14:24.097253 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.097235    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-babea4\"/\"test-model-babea4-kserve-self-signed-certs\""
Nov 03 20:14:24.178941 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.178904    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605b8ced-a357-40a6-8d96-4c051b215128-tls-certs\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.179139 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.178957    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-dshm\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.179207 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.179190    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mc4s\" (UniqueName: \"kubernetes.io/projected/605b8ced-a357-40a6-8d96-4c051b215128-kube-api-access-5mc4s\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.179365 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.179339    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-tmp-dir\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.179470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.179400    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-home\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.179547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.179493    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-model-cache\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.287462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.286582    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-model-cache\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.287462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.286663    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605b8ced-a357-40a6-8d96-4c051b215128-tls-certs\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.287462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.286712    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-dshm\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.287462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.286922    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mc4s\" (UniqueName: \"kubernetes.io/projected/605b8ced-a357-40a6-8d96-4c051b215128-kube-api-access-5mc4s\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.287462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.287050    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-tmp-dir\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.287462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.287077    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-home\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.287957 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.287622    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-home\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.292672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.288235    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-model-cache\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.292672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.288811    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-tmp-dir\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.317562 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.312135    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605b8ced-a357-40a6-8d96-4c051b215128-tls-certs\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.335524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.333802    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-dshm\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.345422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.345391    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mc4s\" (UniqueName: \"kubernetes.io/projected/605b8ced-a357-40a6-8d96-4c051b215128-kube-api-access-5mc4s\") pod \"test-model-babea4-kserve-84fb77dc7-cnqzd\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") " pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.416198 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.416166    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:24.821185 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:24.820620    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"]
Nov 03 20:14:25.432628 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:25.432592    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd" event={"ID":"605b8ced-a357-40a6-8d96-4c051b215128","Type":"ContainerStarted","Data":"a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d"}
Nov 03 20:14:25.433101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:25.432639    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd" event={"ID":"605b8ced-a357-40a6-8d96-4c051b215128","Type":"ContainerStarted","Data":"35affd17a82960d045949e24cfe2fedbe880105cd4b2751bceb58b2d9139fedd"}
Nov 03 20:14:25.452358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:25.452301    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd" podStartSLOduration=1.45228807 podStartE2EDuration="1.45228807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:25.448918716 +0000 UTC m=+2047.961740496" watchObservedRunningTime="2026-11-03 20:14:25.45228807 +0000 UTC m=+2047.965109851"
Nov 03 20:14:26.418456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:26.418276    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:28.799055 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:14:28.798997    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:14:34.417508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:34.417465    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:34.436871 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:34.436841    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:34.528578 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:34.528548    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:14:36.297679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.297632    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"]
Nov 03 20:14:36.305881 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.305856    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.314421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.314387    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2092ba\"/\"default-dockercfg-kl7qg\""
Nov 03 20:14:36.314625 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.314567    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2092ba\"/\"test-model-2092ba-kserve-self-signed-certs\""
Nov 03 20:14:36.316089 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.315955    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2092ba\"/\"kube-root-ca.crt\""
Nov 03 20:14:36.316089 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.316052    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2092ba\"/\"openshift-service-ca.crt\""
Nov 03 20:14:36.317524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.317501    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"]
Nov 03 20:14:36.446420 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.446380    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tmp-dir\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.446711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.446683    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-dshm\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.446848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.446828    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5qlg\" (UniqueName: \"kubernetes.io/projected/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-kube-api-access-v5qlg\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.446917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.446892    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tls-certs\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.447005 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.446988    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-model-cache\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.447150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.447115    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-home\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.558518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.553655    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-dshm\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.565315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.564843    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5qlg\" (UniqueName: \"kubernetes.io/projected/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-kube-api-access-v5qlg\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.565674 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.565654    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tls-certs\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.566511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.566481    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-model-cache\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.567079 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.566975    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-model-cache\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.571706 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.571671    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-home\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.585054 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.572112    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-home\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.585451 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.585396    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tmp-dir\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.586016 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.585895    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tmp-dir\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.592227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.592164    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5qlg\" (UniqueName: \"kubernetes.io/projected/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-kube-api-access-v5qlg\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.613178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.613138    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-dshm\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.622751 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.622683    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tls-certs\") pod \"test-model-2092ba-kserve-69c88c4895-dngbl\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") " pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:36.654666 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:36.654631    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:37.047872 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:14:37.047837    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b68c73f_e9cc_4e95_83ac_5c95a778f6f5.slice/crio-043b22f559a4b4bc02ca0384a3d6d525d88a0222ae91cd32189ef815d57700c2 WatchSource:0}: Error finding container 043b22f559a4b4bc02ca0384a3d6d525d88a0222ae91cd32189ef815d57700c2: Status 404 returned error can't find the container with id 043b22f559a4b4bc02ca0384a3d6d525d88a0222ae91cd32189ef815d57700c2
Nov 03 20:14:37.050260 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:37.050234    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"]
Nov 03 20:14:37.557317 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:37.557283    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl" event={"ID":"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5","Type":"ContainerStarted","Data":"03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7"}
Nov 03 20:14:37.557775 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:37.557325    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl" event={"ID":"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5","Type":"ContainerStarted","Data":"043b22f559a4b4bc02ca0384a3d6d525d88a0222ae91cd32189ef815d57700c2"}
Nov 03 20:14:37.587737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:37.579994    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl" podStartSLOduration=1.579977171 podStartE2EDuration="1.579977171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:37.579262446 +0000 UTC m=+2060.092084230" watchObservedRunningTime="2026-11-03 20:14:37.579977171 +0000 UTC m=+2060.092798954"
Nov 03 20:14:37.656449 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:37.656401    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:46.655999 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:46.655963    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:46.669142 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:46.669113    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:47.669059 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:47.669020    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:14:49.647046 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.647006    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"]
Nov 03 20:14:49.662749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.662714    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.674703 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.674667    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"]
Nov 03 20:14:49.675711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.675683    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn2bl\" (UniqueName: \"kubernetes.io/projected/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-kube-api-access-cn2bl\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.675853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.675744    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-home\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.675853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.675750    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9b1aed\"/\"kube-root-ca.crt\""
Nov 03 20:14:49.675853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.675806    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-model-cache\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.675853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.675843    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9b1aed\"/\"test-model-9b1aed-kserve-self-signed-certs\""
Nov 03 20:14:49.676083 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.675892    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tmp-dir\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.676083 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.675919    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tls-certs\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.676083 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.675939    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9b1aed\"/\"openshift-service-ca.crt\""
Nov 03 20:14:49.676083 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.676023    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9b1aed\"/\"default-dockercfg-74mxq\""
Nov 03 20:14:49.676284 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.676164    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-dshm\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.778256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.778206    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2bl\" (UniqueName: \"kubernetes.io/projected/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-kube-api-access-cn2bl\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.778485 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.778269    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-home\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.778485 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.778318    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-model-cache\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.778485 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.778424    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tmp-dir\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.778485 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.778472    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tls-certs\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.778727 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.778657    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-home\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.778727 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.778681    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-dshm\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.778846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.778777    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-model-cache\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.778905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.778889    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tmp-dir\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.781144 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.781120    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-dshm\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.781472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.781424    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tls-certs\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:49.787567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:49.787542    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn2bl\" (UniqueName: \"kubernetes.io/projected/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-kube-api-access-cn2bl\") pod \"test-model-9b1aed-kserve-796bdc449f-w5ddf\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") " pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:50.012512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:50.007721    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:50.380671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:50.380635    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"]
Nov 03 20:14:50.380998 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:14:50.380968    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod849ff6ad_84b0_4846_8d4a_a31b0b3ee2eb.slice/crio-d453bce951a5d3b154922a963c6de612facbe91fbbeedbf1c03c18aafad91a62 WatchSource:0}: Error finding container d453bce951a5d3b154922a963c6de612facbe91fbbeedbf1c03c18aafad91a62: Status 404 returned error can't find the container with id d453bce951a5d3b154922a963c6de612facbe91fbbeedbf1c03c18aafad91a62
Nov 03 20:14:50.725696 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:50.725654    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf" event={"ID":"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb","Type":"ContainerStarted","Data":"f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6"}
Nov 03 20:14:50.725696 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:50.725711    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf" event={"ID":"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb","Type":"ContainerStarted","Data":"d453bce951a5d3b154922a963c6de612facbe91fbbeedbf1c03c18aafad91a62"}
Nov 03 20:14:50.744005 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:50.743946    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf" podStartSLOduration=1.7439235210000001 podStartE2EDuration="1.743923521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:50.743384588 +0000 UTC m=+2073.256206369" watchObservedRunningTime="2026-11-03 20:14:50.743923521 +0000 UTC m=+2073.256745304"
Nov 03 20:14:51.013827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:14:51.013737    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:14:58.797958 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:14:58.797869    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:15:00.013574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.013524    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:15:00.028165 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.028135    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:15:00.134035 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.134003    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"]
Nov 03 20:15:00.141847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.141808    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"]
Nov 03 20:15:00.142032 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.141920    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"
Nov 03 20:15:00.163914 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.163881    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:15:00.166680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.166652    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"]
Nov 03 20:15:00.170992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.170965    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"
Nov 03 20:15:00.180310 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.180283    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"]
Nov 03 20:15:00.181545 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.180700    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"
Nov 03 20:15:00.196039 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.195705    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"]
Nov 03 20:15:00.196039 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.195759    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"]
Nov 03 20:15:00.196039 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.195789    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"]
Nov 03 20:15:00.196039 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.195821    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"]
Nov 03 20:15:00.196039 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.195899    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"
Nov 03 20:15:00.213277 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.212684    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52gm6\" (UniqueName: \"kubernetes.io/projected/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-kube-api-access-52gm6\") pod \"maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn\" (UID: \"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"
Nov 03 20:15:00.213277 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.213001    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn\" (UID: \"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"
Nov 03 20:15:00.314286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.314184    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmmzt\" (UniqueName: \"kubernetes.io/projected/a1a7697e-c056-445c-93ca-1d1346a48d35-kube-api-access-hmmzt\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj\" (UID: \"a1a7697e-c056-445c-93ca-1d1346a48d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"
Nov 03 20:15:00.314485 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.314334    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn\" (UID: \"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"
Nov 03 20:15:00.314485 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.314399    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9fqx\" (UniqueName: \"kubernetes.io/projected/5835ea77-61e4-452e-8240-ff5ac2d55242-kube-api-access-d9fqx\") pod \"maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk\" (UID: \"5835ea77-61e4-452e-8240-ff5ac2d55242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"
Nov 03 20:15:00.314653 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.314563    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzztq\" (UniqueName: \"kubernetes.io/projected/376c40e6-bb2d-479e-88bf-f9f1c85437b3-kube-api-access-gzztq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c\" (UID: \"376c40e6-bb2d-479e-88bf-f9f1c85437b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"
Nov 03 20:15:00.314653 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.314637    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7697e-c056-445c-93ca-1d1346a48d35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj\" (UID: \"a1a7697e-c056-445c-93ca-1d1346a48d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"
Nov 03 20:15:00.314740 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.314668    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5835ea77-61e4-452e-8240-ff5ac2d55242-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk\" (UID: \"5835ea77-61e4-452e-8240-ff5ac2d55242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"
Nov 03 20:15:00.314740 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.314707    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52gm6\" (UniqueName: \"kubernetes.io/projected/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-kube-api-access-52gm6\") pod \"maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn\" (UID: \"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"
Nov 03 20:15:00.314960 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.314936    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376c40e6-bb2d-479e-88bf-f9f1c85437b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c\" (UID: \"376c40e6-bb2d-479e-88bf-f9f1c85437b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"
Nov 03 20:15:00.315143 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.315117    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn\" (UID: \"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"
Nov 03 20:15:00.323173 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.323144    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52gm6\" (UniqueName: \"kubernetes.io/projected/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-kube-api-access-52gm6\") pod \"maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn\" (UID: \"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"
Nov 03 20:15:00.415974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.415931    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7697e-c056-445c-93ca-1d1346a48d35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj\" (UID: \"a1a7697e-c056-445c-93ca-1d1346a48d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"
Nov 03 20:15:00.415974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.415977    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5835ea77-61e4-452e-8240-ff5ac2d55242-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk\" (UID: \"5835ea77-61e4-452e-8240-ff5ac2d55242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"
Nov 03 20:15:00.416205 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.416081    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376c40e6-bb2d-479e-88bf-f9f1c85437b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c\" (UID: \"376c40e6-bb2d-479e-88bf-f9f1c85437b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"
Nov 03 20:15:00.416205 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.416109    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmmzt\" (UniqueName: \"kubernetes.io/projected/a1a7697e-c056-445c-93ca-1d1346a48d35-kube-api-access-hmmzt\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj\" (UID: \"a1a7697e-c056-445c-93ca-1d1346a48d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"
Nov 03 20:15:00.416306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.416211    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9fqx\" (UniqueName: \"kubernetes.io/projected/5835ea77-61e4-452e-8240-ff5ac2d55242-kube-api-access-d9fqx\") pod \"maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk\" (UID: \"5835ea77-61e4-452e-8240-ff5ac2d55242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"
Nov 03 20:15:00.416306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.416270    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzztq\" (UniqueName: \"kubernetes.io/projected/376c40e6-bb2d-479e-88bf-f9f1c85437b3-kube-api-access-gzztq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c\" (UID: \"376c40e6-bb2d-479e-88bf-f9f1c85437b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"
Nov 03 20:15:00.416867 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.416837    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7697e-c056-445c-93ca-1d1346a48d35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj\" (UID: \"a1a7697e-c056-445c-93ca-1d1346a48d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"
Nov 03 20:15:00.416867 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.416852    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5835ea77-61e4-452e-8240-ff5ac2d55242-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk\" (UID: \"5835ea77-61e4-452e-8240-ff5ac2d55242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"
Nov 03 20:15:00.417005 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.416863    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376c40e6-bb2d-479e-88bf-f9f1c85437b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c\" (UID: \"376c40e6-bb2d-479e-88bf-f9f1c85437b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"
Nov 03 20:15:00.425594 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.425565    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9fqx\" (UniqueName: \"kubernetes.io/projected/5835ea77-61e4-452e-8240-ff5ac2d55242-kube-api-access-d9fqx\") pod \"maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk\" (UID: \"5835ea77-61e4-452e-8240-ff5ac2d55242\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"
Nov 03 20:15:00.425756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.425724    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzztq\" (UniqueName: \"kubernetes.io/projected/376c40e6-bb2d-479e-88bf-f9f1c85437b3-kube-api-access-gzztq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c\" (UID: \"376c40e6-bb2d-479e-88bf-f9f1c85437b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"
Nov 03 20:15:00.425756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.425739    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmmzt\" (UniqueName: \"kubernetes.io/projected/a1a7697e-c056-445c-93ca-1d1346a48d35-kube-api-access-hmmzt\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj\" (UID: \"a1a7697e-c056-445c-93ca-1d1346a48d35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"
Nov 03 20:15:00.497643 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.497606    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"
Nov 03 20:15:00.527179 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.527134    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"
Nov 03 20:15:00.544765 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.544736    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"
Nov 03 20:15:00.565688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.565654    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"
Nov 03 20:15:00.816446 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.816335    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"]
Nov 03 20:15:00.853847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.853800    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"]
Nov 03 20:15:00.855220 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:00.855182    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a7697e_c056_445c_93ca_1d1346a48d35.slice/crio-96483d8a77a7022dbee0e6060afdad9a59988874bd0837f4a0d49b4b70e0d965 WatchSource:0}: Error finding container 96483d8a77a7022dbee0e6060afdad9a59988874bd0837f4a0d49b4b70e0d965: Status 404 returned error can't find the container with id 96483d8a77a7022dbee0e6060afdad9a59988874bd0837f4a0d49b4b70e0d965
Nov 03 20:15:00.862770 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:00.862741    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:15:01.114658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.114230    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"]
Nov 03 20:15:01.118292 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:01.118259    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5835ea77_61e4_452e_8240_ff5ac2d55242.slice/crio-f1b8b515c076c91409e9a825fc591b23782679802fae66c461b375c53b7515d7 WatchSource:0}: Error finding container f1b8b515c076c91409e9a825fc591b23782679802fae66c461b375c53b7515d7: Status 404 returned error can't find the container with id f1b8b515c076c91409e9a825fc591b23782679802fae66c461b375c53b7515d7
Nov 03 20:15:01.128117 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.128085    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"]
Nov 03 20:15:01.140563 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:01.140522    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376c40e6_bb2d_479e_88bf_f9f1c85437b3.slice/crio-d79c4e3e2950110fc0a150dc16628a9de5f3dfc6302f90e37ff9de91fb4e5c36 WatchSource:0}: Error finding container d79c4e3e2950110fc0a150dc16628a9de5f3dfc6302f90e37ff9de91fb4e5c36: Status 404 returned error can't find the container with id d79c4e3e2950110fc0a150dc16628a9de5f3dfc6302f90e37ff9de91fb4e5c36
Nov 03 20:15:01.852248 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.852133    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk" event={"ID":"5835ea77-61e4-452e-8240-ff5ac2d55242","Type":"ContainerStarted","Data":"78bf1c651443457c109b291ca1cb68ecf5d7f2f60f151e1d49d4713d1b6f9814"}
Nov 03 20:15:01.852248 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.852197    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk" event={"ID":"5835ea77-61e4-452e-8240-ff5ac2d55242","Type":"ContainerStarted","Data":"f1b8b515c076c91409e9a825fc591b23782679802fae66c461b375c53b7515d7"}
Nov 03 20:15:01.854051 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.854017    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj" event={"ID":"a1a7697e-c056-445c-93ca-1d1346a48d35","Type":"ContainerStarted","Data":"3b073ea4f048bb1597eeb068b62f2d5ddc3e4736e11b77f9117a927176b2e67c"}
Nov 03 20:15:01.854165 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.854072    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj" event={"ID":"a1a7697e-c056-445c-93ca-1d1346a48d35","Type":"ContainerStarted","Data":"96483d8a77a7022dbee0e6060afdad9a59988874bd0837f4a0d49b4b70e0d965"}
Nov 03 20:15:01.856182 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.856157    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn" event={"ID":"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2","Type":"ContainerStarted","Data":"82f1e84317e307ce9952aea2ab5993f46ee9818a8f3ffea80b77865271fc5e2a"}
Nov 03 20:15:01.856300 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.856205    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn" event={"ID":"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2","Type":"ContainerStarted","Data":"d47f39d510602b624df9cbc1c0cd1e5302b7af99ed2674dfd5556911f39c1743"}
Nov 03 20:15:01.857988 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.857954    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" event={"ID":"376c40e6-bb2d-479e-88bf-f9f1c85437b3","Type":"ContainerStarted","Data":"125b45709d5e848844ce30fb8f2ead15a8243c4576daf03c2fee1c17a0dc3f09"}
Nov 03 20:15:01.858074 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.858002    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" event={"ID":"376c40e6-bb2d-479e-88bf-f9f1c85437b3","Type":"ContainerStarted","Data":"d79c4e3e2950110fc0a150dc16628a9de5f3dfc6302f90e37ff9de91fb4e5c36"}
Nov 03 20:15:01.871311 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.871252    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk" podStartSLOduration=1.871234452 podStartE2EDuration="1.871234452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:01.86840201 +0000 UTC m=+2084.381223803" watchObservedRunningTime="2026-11-03 20:15:01.871234452 +0000 UTC m=+2084.384056235"
Nov 03 20:15:01.889758 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.889691    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj" podStartSLOduration=1.889674473 podStartE2EDuration="1.889674473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:01.88567338 +0000 UTC m=+2084.398495164" watchObservedRunningTime="2026-11-03 20:15:01.889674473 +0000 UTC m=+2084.402496257"
Nov 03 20:15:01.903695 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.903638    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn" podStartSLOduration=1.9036251320000002 podStartE2EDuration="1.903625132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:01.901291137 +0000 UTC m=+2084.414112919" watchObservedRunningTime="2026-11-03 20:15:01.903625132 +0000 UTC m=+2084.416446947"
Nov 03 20:15:01.922106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:01.922046    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" podStartSLOduration=1.9220322570000001 podStartE2EDuration="1.922032257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:01.9180756 +0000 UTC m=+2084.430897385" watchObservedRunningTime="2026-11-03 20:15:01.922032257 +0000 UTC m=+2084.434854040"
Nov 03 20:15:02.868047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.868013    5600 generic.go:356] "Generic (PLEG): container finished" podID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerID="125b45709d5e848844ce30fb8f2ead15a8243c4576daf03c2fee1c17a0dc3f09" exitCode=60
Nov 03 20:15:02.868531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.868090    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" event={"ID":"376c40e6-bb2d-479e-88bf-f9f1c85437b3","Type":"ContainerDied","Data":"125b45709d5e848844ce30fb8f2ead15a8243c4576daf03c2fee1c17a0dc3f09"}
Nov 03 20:15:02.868531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.868491    5600 scope.go:122] "RemoveContainer" containerID="125b45709d5e848844ce30fb8f2ead15a8243c4576daf03c2fee1c17a0dc3f09"
Nov 03 20:15:02.870040 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.870016    5600 generic.go:356] "Generic (PLEG): container finished" podID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerID="78bf1c651443457c109b291ca1cb68ecf5d7f2f60f151e1d49d4713d1b6f9814" exitCode=60
Nov 03 20:15:02.870172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.870095    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk" event={"ID":"5835ea77-61e4-452e-8240-ff5ac2d55242","Type":"ContainerDied","Data":"78bf1c651443457c109b291ca1cb68ecf5d7f2f60f151e1d49d4713d1b6f9814"}
Nov 03 20:15:02.870546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.870383    5600 scope.go:122] "RemoveContainer" containerID="78bf1c651443457c109b291ca1cb68ecf5d7f2f60f151e1d49d4713d1b6f9814"
Nov 03 20:15:02.872075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.872051    5600 generic.go:356] "Generic (PLEG): container finished" podID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerID="3b073ea4f048bb1597eeb068b62f2d5ddc3e4736e11b77f9117a927176b2e67c" exitCode=60
Nov 03 20:15:02.872250 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.872132    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj" event={"ID":"a1a7697e-c056-445c-93ca-1d1346a48d35","Type":"ContainerDied","Data":"3b073ea4f048bb1597eeb068b62f2d5ddc3e4736e11b77f9117a927176b2e67c"}
Nov 03 20:15:02.872466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.872424    5600 scope.go:122] "RemoveContainer" containerID="3b073ea4f048bb1597eeb068b62f2d5ddc3e4736e11b77f9117a927176b2e67c"
Nov 03 20:15:02.874390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.874292    5600 generic.go:356] "Generic (PLEG): container finished" podID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerID="82f1e84317e307ce9952aea2ab5993f46ee9818a8f3ffea80b77865271fc5e2a" exitCode=60
Nov 03 20:15:02.874390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.874375    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn" event={"ID":"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2","Type":"ContainerDied","Data":"82f1e84317e307ce9952aea2ab5993f46ee9818a8f3ffea80b77865271fc5e2a"}
Nov 03 20:15:02.874697 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:02.874678    5600 scope.go:122] "RemoveContainer" containerID="82f1e84317e307ce9952aea2ab5993f46ee9818a8f3ffea80b77865271fc5e2a"
Nov 03 20:15:03.889904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:03.889865    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk" event={"ID":"5835ea77-61e4-452e-8240-ff5ac2d55242","Type":"ContainerStarted","Data":"b6b9127cf4b7a06113bf246c1f37a812c36aff0dc046c51dae1af1ee5ab88192"}
Nov 03 20:15:03.893195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:03.893117    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj" event={"ID":"a1a7697e-c056-445c-93ca-1d1346a48d35","Type":"ContainerStarted","Data":"7e3c35623481b880d5f46fe3d246ae6665cf764f6990b714441578b6b97756dc"}
Nov 03 20:15:03.896833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:03.896804    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn" event={"ID":"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2","Type":"ContainerStarted","Data":"1242baa095de38c661581ce5fe18e867e22f9dd04b1a03b009f73b3321c95561"}
Nov 03 20:15:03.920663 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:03.920628    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" event={"ID":"376c40e6-bb2d-479e-88bf-f9f1c85437b3","Type":"ContainerStarted","Data":"9907d2c2c11773151f69a2b34ab3055042848a0d87f4f23c55ab5fa76a9aed4e"}
Nov 03 20:15:04.931621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.931585    5600 generic.go:356] "Generic (PLEG): container finished" podID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerID="9907d2c2c11773151f69a2b34ab3055042848a0d87f4f23c55ab5fa76a9aed4e" exitCode=60
Nov 03 20:15:04.932076 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.931669    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" event={"ID":"376c40e6-bb2d-479e-88bf-f9f1c85437b3","Type":"ContainerDied","Data":"9907d2c2c11773151f69a2b34ab3055042848a0d87f4f23c55ab5fa76a9aed4e"}
Nov 03 20:15:04.932076 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.931732    5600 scope.go:122] "RemoveContainer" containerID="125b45709d5e848844ce30fb8f2ead15a8243c4576daf03c2fee1c17a0dc3f09"
Nov 03 20:15:04.932184 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.932098    5600 scope.go:122] "RemoveContainer" containerID="9907d2c2c11773151f69a2b34ab3055042848a0d87f4f23c55ab5fa76a9aed4e"
Nov 03 20:15:04.932608 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:04.932579    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c_odh-ai-gateway-infra(376c40e6-bb2d-479e-88bf-f9f1c85437b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3"
Nov 03 20:15:04.934156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.934117    5600 generic.go:356] "Generic (PLEG): container finished" podID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerID="b6b9127cf4b7a06113bf246c1f37a812c36aff0dc046c51dae1af1ee5ab88192" exitCode=60
Nov 03 20:15:04.934261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.934181    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk" event={"ID":"5835ea77-61e4-452e-8240-ff5ac2d55242","Type":"ContainerDied","Data":"b6b9127cf4b7a06113bf246c1f37a812c36aff0dc046c51dae1af1ee5ab88192"}
Nov 03 20:15:04.934609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.934590    5600 scope.go:122] "RemoveContainer" containerID="b6b9127cf4b7a06113bf246c1f37a812c36aff0dc046c51dae1af1ee5ab88192"
Nov 03 20:15:04.935059 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:04.935036    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk_odh-ai-gateway-infra(5835ea77-61e4-452e-8240-ff5ac2d55242)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242"
Nov 03 20:15:04.936544 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.936525    5600 generic.go:356] "Generic (PLEG): container finished" podID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerID="7e3c35623481b880d5f46fe3d246ae6665cf764f6990b714441578b6b97756dc" exitCode=60
Nov 03 20:15:04.936660 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.936603    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj" event={"ID":"a1a7697e-c056-445c-93ca-1d1346a48d35","Type":"ContainerDied","Data":"7e3c35623481b880d5f46fe3d246ae6665cf764f6990b714441578b6b97756dc"}
Nov 03 20:15:04.936848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.936831    5600 scope.go:122] "RemoveContainer" containerID="7e3c35623481b880d5f46fe3d246ae6665cf764f6990b714441578b6b97756dc"
Nov 03 20:15:04.937243 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:04.937196    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj_odh-ai-gateway-infra(a1a7697e-c056-445c-93ca-1d1346a48d35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35"
Nov 03 20:15:04.938724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.938707    5600 generic.go:356] "Generic (PLEG): container finished" podID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerID="1242baa095de38c661581ce5fe18e867e22f9dd04b1a03b009f73b3321c95561" exitCode=60
Nov 03 20:15:04.938811 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.938775    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn" event={"ID":"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2","Type":"ContainerDied","Data":"1242baa095de38c661581ce5fe18e867e22f9dd04b1a03b009f73b3321c95561"}
Nov 03 20:15:04.938980 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.938967    5600 scope.go:122] "RemoveContainer" containerID="1242baa095de38c661581ce5fe18e867e22f9dd04b1a03b009f73b3321c95561"
Nov 03 20:15:04.939208 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:04.939191    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn_odh-ai-gateway-infra(c6eebfb3-3269-4ae2-b530-6d5680c6b3d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2"
Nov 03 20:15:04.946575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.946553    5600 scope.go:122] "RemoveContainer" containerID="78bf1c651443457c109b291ca1cb68ecf5d7f2f60f151e1d49d4713d1b6f9814"
Nov 03 20:15:04.961266 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.961245    5600 scope.go:122] "RemoveContainer" containerID="3b073ea4f048bb1597eeb068b62f2d5ddc3e4736e11b77f9117a927176b2e67c"
Nov 03 20:15:04.976370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:04.976348    5600 scope.go:122] "RemoveContainer" containerID="82f1e84317e307ce9952aea2ab5993f46ee9818a8f3ffea80b77865271fc5e2a"
Nov 03 20:15:05.950388 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:05.950352    5600 scope.go:122] "RemoveContainer" containerID="b6b9127cf4b7a06113bf246c1f37a812c36aff0dc046c51dae1af1ee5ab88192"
Nov 03 20:15:05.951128 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:05.950841    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk_odh-ai-gateway-infra(5835ea77-61e4-452e-8240-ff5ac2d55242)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242"
Nov 03 20:15:05.953535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:05.953516    5600 scope.go:122] "RemoveContainer" containerID="7e3c35623481b880d5f46fe3d246ae6665cf764f6990b714441578b6b97756dc"
Nov 03 20:15:05.954197 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:05.954171    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj_odh-ai-gateway-infra(a1a7697e-c056-445c-93ca-1d1346a48d35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35"
Nov 03 20:15:05.958368 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:05.958347    5600 scope.go:122] "RemoveContainer" containerID="1242baa095de38c661581ce5fe18e867e22f9dd04b1a03b009f73b3321c95561"
Nov 03 20:15:05.959549 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:05.958876    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn_odh-ai-gateway-infra(c6eebfb3-3269-4ae2-b530-6d5680c6b3d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2"
Nov 03 20:15:05.961376 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:05.961356    5600 scope.go:122] "RemoveContainer" containerID="9907d2c2c11773151f69a2b34ab3055042848a0d87f4f23c55ab5fa76a9aed4e"
Nov 03 20:15:05.961765 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:05.961741    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c_odh-ai-gateway-infra(376c40e6-bb2d-479e-88bf-f9f1c85437b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3"
Nov 03 20:15:14.609779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:14.609745    5600 scope.go:122] "RemoveContainer" containerID="9907d2c2c11773151f69a2b34ab3055042848a0d87f4f23c55ab5fa76a9aed4e"
Nov 03 20:15:15.045505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:15.045473    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" event={"ID":"376c40e6-bb2d-479e-88bf-f9f1c85437b3","Type":"ContainerStarted","Data":"6cf4fdd7a9032393913646c5cd9118b70daf803298673b096be831aeef8c109f"}
Nov 03 20:15:16.055221 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:16.055139    5600 generic.go:356] "Generic (PLEG): container finished" podID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerID="6cf4fdd7a9032393913646c5cd9118b70daf803298673b096be831aeef8c109f" exitCode=60
Nov 03 20:15:16.055221 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:16.055212    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" event={"ID":"376c40e6-bb2d-479e-88bf-f9f1c85437b3","Type":"ContainerDied","Data":"6cf4fdd7a9032393913646c5cd9118b70daf803298673b096be831aeef8c109f"}
Nov 03 20:15:16.055705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:16.055277    5600 scope.go:122] "RemoveContainer" containerID="9907d2c2c11773151f69a2b34ab3055042848a0d87f4f23c55ab5fa76a9aed4e"
Nov 03 20:15:16.055868 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:16.055822    5600 scope.go:122] "RemoveContainer" containerID="6cf4fdd7a9032393913646c5cd9118b70daf803298673b096be831aeef8c109f"
Nov 03 20:15:16.056276 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:16.056252    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c_odh-ai-gateway-infra(376c40e6-bb2d-479e-88bf-f9f1c85437b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3"
Nov 03 20:15:16.123171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:16.123123    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"]
Nov 03 20:15:17.518490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.518402    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"
Nov 03 20:15:17.552395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.552360    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376c40e6-bb2d-479e-88bf-f9f1c85437b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376c40e6-bb2d-479e-88bf-f9f1c85437b3-maas-api-service-ca\") pod \"376c40e6-bb2d-479e-88bf-f9f1c85437b3\" (UID: \"376c40e6-bb2d-479e-88bf-f9f1c85437b3\") "
Nov 03 20:15:17.552657 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.552498    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376c40e6-bb2d-479e-88bf-f9f1c85437b3-kube-api-access-gzztq\" (UniqueName: \"kubernetes.io/projected/376c40e6-bb2d-479e-88bf-f9f1c85437b3-kube-api-access-gzztq\") pod \"376c40e6-bb2d-479e-88bf-f9f1c85437b3\" (UID: \"376c40e6-bb2d-479e-88bf-f9f1c85437b3\") "
Nov 03 20:15:17.554767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.552956    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376c40e6-bb2d-479e-88bf-f9f1c85437b3-maas-api-service-ca" pod "376c40e6-bb2d-479e-88bf-f9f1c85437b3" (UID: "376c40e6-bb2d-479e-88bf-f9f1c85437b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:17.554767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.553616    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376c40e6-bb2d-479e-88bf-f9f1c85437b3-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:17.557947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.557913    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376c40e6-bb2d-479e-88bf-f9f1c85437b3-kube-api-access-gzztq" pod "376c40e6-bb2d-479e-88bf-f9f1c85437b3" (UID: "376c40e6-bb2d-479e-88bf-f9f1c85437b3"). InnerVolumeSpecName "kube-api-access-gzztq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:17.559872 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.559842    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"]
Nov 03 20:15:17.561639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.561093    5600 cpu_manager.go:414] "containerMap: removing container" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerName="cleanup"
Nov 03 20:15:17.561639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.561115    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerName="cleanup"
Nov 03 20:15:17.561639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.561151    5600 cpu_manager.go:414] "containerMap: removing container" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerName="cleanup"
Nov 03 20:15:17.561639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.561160    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerName="cleanup"
Nov 03 20:15:17.561639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.561172    5600 cpu_manager.go:414] "containerMap: removing container" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerName="cleanup"
Nov 03 20:15:17.561639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.561179    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerName="cleanup"
Nov 03 20:15:17.561639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.561351    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerName="cleanup"
Nov 03 20:15:17.561639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.561369    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerName="cleanup"
Nov 03 20:15:17.561639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.561377    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" containerName="cleanup"
Nov 03 20:15:17.575366 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.565369    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"
Nov 03 20:15:17.575366 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.569921    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"]
Nov 03 20:15:17.629651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.616007    5600 scope.go:122] "RemoveContainer" containerID="7e3c35623481b880d5f46fe3d246ae6665cf764f6990b714441578b6b97756dc"
Nov 03 20:15:17.644956 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.644843    5600 scope.go:122] "RemoveContainer" containerID="b6b9127cf4b7a06113bf246c1f37a812c36aff0dc046c51dae1af1ee5ab88192"
Nov 03 20:15:17.656005 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.655970    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfjx7\" (UniqueName: \"kubernetes.io/projected/ce12762c-ce10-424d-baed-f43ab92bef20-kube-api-access-wfjx7\") pod \"maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k\" (UID: \"ce12762c-ce10-424d-baed-f43ab92bef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"
Nov 03 20:15:17.656187 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.656060    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce12762c-ce10-424d-baed-f43ab92bef20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k\" (UID: \"ce12762c-ce10-424d-baed-f43ab92bef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"
Nov 03 20:15:17.656187 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.656175    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzztq\" (UniqueName: \"kubernetes.io/projected/376c40e6-bb2d-479e-88bf-f9f1c85437b3-kube-api-access-gzztq\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:17.758017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.757976    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfjx7\" (UniqueName: \"kubernetes.io/projected/ce12762c-ce10-424d-baed-f43ab92bef20-kube-api-access-wfjx7\") pod \"maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k\" (UID: \"ce12762c-ce10-424d-baed-f43ab92bef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"
Nov 03 20:15:17.758236 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.758068    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce12762c-ce10-424d-baed-f43ab92bef20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k\" (UID: \"ce12762c-ce10-424d-baed-f43ab92bef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"
Nov 03 20:15:17.758918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.758893    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce12762c-ce10-424d-baed-f43ab92bef20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k\" (UID: \"ce12762c-ce10-424d-baed-f43ab92bef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"
Nov 03 20:15:17.768542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.768503    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfjx7\" (UniqueName: \"kubernetes.io/projected/ce12762c-ce10-424d-baed-f43ab92bef20-kube-api-access-wfjx7\") pod \"maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k\" (UID: \"ce12762c-ce10-424d-baed-f43ab92bef20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"
Nov 03 20:15:17.937155 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:17.937112    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"
Nov 03 20:15:18.138028 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:18.137997    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"
Nov 03 20:15:18.138247 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:18.138028    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c" event={"ID":"376c40e6-bb2d-479e-88bf-f9f1c85437b3","Type":"ContainerDied","Data":"d79c4e3e2950110fc0a150dc16628a9de5f3dfc6302f90e37ff9de91fb4e5c36"}
Nov 03 20:15:18.138247 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:18.138105    5600 scope.go:122] "RemoveContainer" containerID="6cf4fdd7a9032393913646c5cd9118b70daf803298673b096be831aeef8c109f"
Nov 03 20:15:18.158541 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:18.158512    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"]
Nov 03 20:15:18.169274 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:18.169240    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-944e83-29895615-49x2c"]
Nov 03 20:15:18.235974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:18.235945    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"]
Nov 03 20:15:18.236396 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:18.236375    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce12762c_ce10_424d_baed_f43ab92bef20.slice/crio-24901a32e5db03af7fb642b3129a25e445a542e54020fcc1f45fa257e6195baf WatchSource:0}: Error finding container 24901a32e5db03af7fb642b3129a25e445a542e54020fcc1f45fa257e6195baf: Status 404 returned error can't find the container with id 24901a32e5db03af7fb642b3129a25e445a542e54020fcc1f45fa257e6195baf
Nov 03 20:15:18.627351 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:18.615875    5600 scope.go:122] "RemoveContainer" containerID="1242baa095de38c661581ce5fe18e867e22f9dd04b1a03b009f73b3321c95561"
Nov 03 20:15:19.148322 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.148282    5600 generic.go:356] "Generic (PLEG): container finished" podID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerID="7adf23cd89e75c02abc72e1ff8e365f67b5d1ae1ea47ed71c95d893fbe5734c6" exitCode=60
Nov 03 20:15:19.148583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.148353    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk" event={"ID":"5835ea77-61e4-452e-8240-ff5ac2d55242","Type":"ContainerDied","Data":"7adf23cd89e75c02abc72e1ff8e365f67b5d1ae1ea47ed71c95d893fbe5734c6"}
Nov 03 20:15:19.148583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.148497    5600 scope.go:122] "RemoveContainer" containerID="b6b9127cf4b7a06113bf246c1f37a812c36aff0dc046c51dae1af1ee5ab88192"
Nov 03 20:15:19.149013 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.148989    5600 scope.go:122] "RemoveContainer" containerID="7adf23cd89e75c02abc72e1ff8e365f67b5d1ae1ea47ed71c95d893fbe5734c6"
Nov 03 20:15:19.150798 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:19.149421    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk_odh-ai-gateway-infra(5835ea77-61e4-452e-8240-ff5ac2d55242)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242"
Nov 03 20:15:19.150798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.150426    5600 generic.go:356] "Generic (PLEG): container finished" podID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerID="3a4cd50491e36d54fa7d4dc72b7a72a71f65056864cac1051b986aa44bf61495" exitCode=60
Nov 03 20:15:19.150798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.150465    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj" event={"ID":"a1a7697e-c056-445c-93ca-1d1346a48d35","Type":"ContainerDied","Data":"3a4cd50491e36d54fa7d4dc72b7a72a71f65056864cac1051b986aa44bf61495"}
Nov 03 20:15:19.150916 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.150872    5600 scope.go:122] "RemoveContainer" containerID="3a4cd50491e36d54fa7d4dc72b7a72a71f65056864cac1051b986aa44bf61495"
Nov 03 20:15:19.151270 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:19.151244    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj_odh-ai-gateway-infra(a1a7697e-c056-445c-93ca-1d1346a48d35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35"
Nov 03 20:15:19.152881 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.152857    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn" event={"ID":"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2","Type":"ContainerStarted","Data":"76aa34276dfd7240dc75799446e89404ccd96bf7d2f8099c1febb4dc7bddbc96"}
Nov 03 20:15:19.155873 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.155844    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k" event={"ID":"ce12762c-ce10-424d-baed-f43ab92bef20","Type":"ContainerStarted","Data":"2691fbd292eebe0807b31defd9f700fcc0eab8d3987c4ba004d14d63ad5814b7"}
Nov 03 20:15:19.156005 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.155897    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k" event={"ID":"ce12762c-ce10-424d-baed-f43ab92bef20","Type":"ContainerStarted","Data":"24901a32e5db03af7fb642b3129a25e445a542e54020fcc1f45fa257e6195baf"}
Nov 03 20:15:19.163376 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.163354    5600 scope.go:122] "RemoveContainer" containerID="7e3c35623481b880d5f46fe3d246ae6665cf764f6990b714441578b6b97756dc"
Nov 03 20:15:19.205469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.203318    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k" podStartSLOduration=19.203294785 podStartE2EDuration="19.203294785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:19.192993017 +0000 UTC m=+2101.705814799" watchObservedRunningTime="2026-11-03 20:15:19.203294785 +0000 UTC m=+2101.716116568"
Nov 03 20:15:19.625087 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:19.624989    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376c40e6-bb2d-479e-88bf-f9f1c85437b3" path="/var/lib/kubelet/pods/376c40e6-bb2d-479e-88bf-f9f1c85437b3/volumes"
Nov 03 20:15:20.262045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.249855    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"]
Nov 03 20:15:20.262045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.257091    5600 generic.go:356] "Generic (PLEG): container finished" podID="ce12762c-ce10-424d-baed-f43ab92bef20" containerID="2691fbd292eebe0807b31defd9f700fcc0eab8d3987c4ba004d14d63ad5814b7" exitCode=60
Nov 03 20:15:20.262045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.257361    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k" event={"ID":"ce12762c-ce10-424d-baed-f43ab92bef20","Type":"ContainerDied","Data":"2691fbd292eebe0807b31defd9f700fcc0eab8d3987c4ba004d14d63ad5814b7"}
Nov 03 20:15:20.262045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.257937    5600 scope.go:122] "RemoveContainer" containerID="2691fbd292eebe0807b31defd9f700fcc0eab8d3987c4ba004d14d63ad5814b7"
Nov 03 20:15:20.330493 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.329175    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"]
Nov 03 20:15:20.354758 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.354720    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"]
Nov 03 20:15:20.357177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.357033    5600 generic.go:356] "Generic (PLEG): container finished" podID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerID="76aa34276dfd7240dc75799446e89404ccd96bf7d2f8099c1febb4dc7bddbc96" exitCode=60
Nov 03 20:15:20.357177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.357119    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn" event={"ID":"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2","Type":"ContainerDied","Data":"76aa34276dfd7240dc75799446e89404ccd96bf7d2f8099c1febb4dc7bddbc96"}
Nov 03 20:15:20.357177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.357169    5600 scope.go:122] "RemoveContainer" containerID="1242baa095de38c661581ce5fe18e867e22f9dd04b1a03b009f73b3321c95561"
Nov 03 20:15:20.863660 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.861349    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"
Nov 03 20:15:20.879532 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.877258    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5835ea77-61e4-452e-8240-ff5ac2d55242-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5835ea77-61e4-452e-8240-ff5ac2d55242-maas-api-service-ca\") pod \"5835ea77-61e4-452e-8240-ff5ac2d55242\" (UID: \"5835ea77-61e4-452e-8240-ff5ac2d55242\") "
Nov 03 20:15:20.879532 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.877523    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5835ea77-61e4-452e-8240-ff5ac2d55242-kube-api-access-d9fqx\" (UniqueName: \"kubernetes.io/projected/5835ea77-61e4-452e-8240-ff5ac2d55242-kube-api-access-d9fqx\") pod \"5835ea77-61e4-452e-8240-ff5ac2d55242\" (UID: \"5835ea77-61e4-452e-8240-ff5ac2d55242\") "
Nov 03 20:15:20.909834 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.896748    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5835ea77-61e4-452e-8240-ff5ac2d55242-maas-api-service-ca" pod "5835ea77-61e4-452e-8240-ff5ac2d55242" (UID: "5835ea77-61e4-452e-8240-ff5ac2d55242"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:20.941947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.937510    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"]
Nov 03 20:15:20.941947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.938697    5600 cpu_manager.go:414] "containerMap: removing container" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerName="cleanup"
Nov 03 20:15:20.941947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.938719    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerName="cleanup"
Nov 03 20:15:20.941947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.938778    5600 cpu_manager.go:414] "containerMap: removing container" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerName="cleanup"
Nov 03 20:15:20.941947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.938786    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerName="cleanup"
Nov 03 20:15:20.941947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.938960    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerName="cleanup"
Nov 03 20:15:20.941947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.938979    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerName="cleanup"
Nov 03 20:15:20.944422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.944384    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5835ea77-61e4-452e-8240-ff5ac2d55242-kube-api-access-d9fqx" pod "5835ea77-61e4-452e-8240-ff5ac2d55242" (UID: "5835ea77-61e4-452e-8240-ff5ac2d55242"). InnerVolumeSpecName "kube-api-access-d9fqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:20.945166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.945144    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"]
Nov 03 20:15:20.945297 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:20.945272    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"
Nov 03 20:15:21.008205 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.008169    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvq7k\" (UniqueName: \"kubernetes.io/projected/a10bb390-2891-4be6-9550-f5dbfff9a352-kube-api-access-pvq7k\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs\" (UID: \"a10bb390-2891-4be6-9550-f5dbfff9a352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"
Nov 03 20:15:21.008663 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.008635    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10bb390-2891-4be6-9550-f5dbfff9a352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs\" (UID: \"a10bb390-2891-4be6-9550-f5dbfff9a352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"
Nov 03 20:15:21.008959 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.008942    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5835ea77-61e4-452e-8240-ff5ac2d55242-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:21.009057 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.009045    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9fqx\" (UniqueName: \"kubernetes.io/projected/5835ea77-61e4-452e-8240-ff5ac2d55242-kube-api-access-d9fqx\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:21.074734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.074708    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"
Nov 03 20:15:21.077383 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.077363    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"
Nov 03 20:15:21.124571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.123255    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a7697e-c056-445c-93ca-1d1346a48d35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7697e-c056-445c-93ca-1d1346a48d35-maas-api-service-ca\") pod \"a1a7697e-c056-445c-93ca-1d1346a48d35\" (UID: \"a1a7697e-c056-445c-93ca-1d1346a48d35\") "
Nov 03 20:15:21.124571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.123499    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a7697e-c056-445c-93ca-1d1346a48d35-kube-api-access-hmmzt\" (UniqueName: \"kubernetes.io/projected/a1a7697e-c056-445c-93ca-1d1346a48d35-kube-api-access-hmmzt\") pod \"a1a7697e-c056-445c-93ca-1d1346a48d35\" (UID: \"a1a7697e-c056-445c-93ca-1d1346a48d35\") "
Nov 03 20:15:21.124571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.123998    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10bb390-2891-4be6-9550-f5dbfff9a352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs\" (UID: \"a10bb390-2891-4be6-9550-f5dbfff9a352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"
Nov 03 20:15:21.124571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.124257    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvq7k\" (UniqueName: \"kubernetes.io/projected/a10bb390-2891-4be6-9550-f5dbfff9a352-kube-api-access-pvq7k\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs\" (UID: \"a10bb390-2891-4be6-9550-f5dbfff9a352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"
Nov 03 20:15:21.128512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.126173    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10bb390-2891-4be6-9550-f5dbfff9a352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs\" (UID: \"a10bb390-2891-4be6-9550-f5dbfff9a352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"
Nov 03 20:15:21.128512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.126997    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a7697e-c056-445c-93ca-1d1346a48d35-maas-api-service-ca" pod "a1a7697e-c056-445c-93ca-1d1346a48d35" (UID: "a1a7697e-c056-445c-93ca-1d1346a48d35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:21.128512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.127882    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a7697e-c056-445c-93ca-1d1346a48d35-kube-api-access-hmmzt" pod "a1a7697e-c056-445c-93ca-1d1346a48d35" (UID: "a1a7697e-c056-445c-93ca-1d1346a48d35"). InnerVolumeSpecName "kube-api-access-hmmzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.133032    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"]
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134236    5600 cpu_manager.go:414] "containerMap: removing container" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134256    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134282    5600 cpu_manager.go:414] "containerMap: removing container" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134290    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134309    5600 cpu_manager.go:414] "containerMap: removing container" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134318    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134330    5600 cpu_manager.go:414] "containerMap: removing container" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134338    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134357    5600 cpu_manager.go:414] "containerMap: removing container" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134365    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134376    5600 cpu_manager.go:414] "containerMap: removing container" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134383    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134405    5600 cpu_manager.go:414] "containerMap: removing container" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerName="cleanup"
Nov 03 20:15:21.137556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.134412    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerName="cleanup"
Nov 03 20:15:21.151012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.143360    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerName="cleanup"
Nov 03 20:15:21.151012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.143391    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerName="cleanup"
Nov 03 20:15:21.151012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.143400    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242" containerName="cleanup"
Nov 03 20:15:21.151012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.143410    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerName="cleanup"
Nov 03 20:15:21.151012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.143459    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerName="cleanup"
Nov 03 20:15:21.151012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.143471    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" containerName="cleanup"
Nov 03 20:15:21.162864 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.162829    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"]
Nov 03 20:15:21.163384 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.163364    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"
Nov 03 20:15:21.163815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.163786    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvq7k\" (UniqueName: \"kubernetes.io/projected/a10bb390-2891-4be6-9550-f5dbfff9a352-kube-api-access-pvq7k\") pod \"maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs\" (UID: \"a10bb390-2891-4be6-9550-f5dbfff9a352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"
Nov 03 20:15:21.164146 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.164124    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35" containerName="cleanup"
Nov 03 20:15:21.169638 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.169615    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"]
Nov 03 20:15:21.169769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.169655    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"]
Nov 03 20:15:21.169769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.169745    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"
Nov 03 20:15:21.225984 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.225946    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-kube-api-access-52gm6\" (UniqueName: \"kubernetes.io/projected/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-kube-api-access-52gm6\") pod \"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2\" (UID: \"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2\") "
Nov 03 20:15:21.226170 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.226116    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-maas-api-service-ca\") pod \"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2\" (UID: \"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2\") "
Nov 03 20:15:21.226728 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.226698    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnjqn\" (UniqueName: \"kubernetes.io/projected/36123290-c071-43cd-8731-a44a4d12597b-kube-api-access-gnjqn\") pod \"maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz\" (UID: \"36123290-c071-43cd-8731-a44a4d12597b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"
Nov 03 20:15:21.226890 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.226871    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6411bde7-2df7-4716-af46-938a3a8ce0bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9\" (UID: \"6411bde7-2df7-4716-af46-938a3a8ce0bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"
Nov 03 20:15:21.227000 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.226884    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-maas-api-service-ca" pod "c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" (UID: "c6eebfb3-3269-4ae2-b530-6d5680c6b3d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:21.227081 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.227062    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kldb\" (UniqueName: \"kubernetes.io/projected/6411bde7-2df7-4716-af46-938a3a8ce0bc-kube-api-access-8kldb\") pod \"maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9\" (UID: \"6411bde7-2df7-4716-af46-938a3a8ce0bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"
Nov 03 20:15:21.227267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.227244    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36123290-c071-43cd-8731-a44a4d12597b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz\" (UID: \"36123290-c071-43cd-8731-a44a4d12597b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"
Nov 03 20:15:21.227617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.227596    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmmzt\" (UniqueName: \"kubernetes.io/projected/a1a7697e-c056-445c-93ca-1d1346a48d35-kube-api-access-hmmzt\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:21.227731 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.227718    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:21.227839 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.227826    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7697e-c056-445c-93ca-1d1346a48d35-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:21.229735 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.229710    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-kube-api-access-52gm6" pod "c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" (UID: "c6eebfb3-3269-4ae2-b530-6d5680c6b3d2"). InnerVolumeSpecName "kube-api-access-52gm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:21.300568 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.300526    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"
Nov 03 20:15:21.355458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.336511    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnjqn\" (UniqueName: \"kubernetes.io/projected/36123290-c071-43cd-8731-a44a4d12597b-kube-api-access-gnjqn\") pod \"maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz\" (UID: \"36123290-c071-43cd-8731-a44a4d12597b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"
Nov 03 20:15:21.355458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.336695    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6411bde7-2df7-4716-af46-938a3a8ce0bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9\" (UID: \"6411bde7-2df7-4716-af46-938a3a8ce0bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"
Nov 03 20:15:21.355458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.336816    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kldb\" (UniqueName: \"kubernetes.io/projected/6411bde7-2df7-4716-af46-938a3a8ce0bc-kube-api-access-8kldb\") pod \"maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9\" (UID: \"6411bde7-2df7-4716-af46-938a3a8ce0bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"
Nov 03 20:15:21.355458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.336857    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36123290-c071-43cd-8731-a44a4d12597b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz\" (UID: \"36123290-c071-43cd-8731-a44a4d12597b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"
Nov 03 20:15:21.355458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.337076    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52gm6\" (UniqueName: \"kubernetes.io/projected/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2-kube-api-access-52gm6\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:21.355458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.348330    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36123290-c071-43cd-8731-a44a4d12597b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz\" (UID: \"36123290-c071-43cd-8731-a44a4d12597b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"
Nov 03 20:15:21.355458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.353981    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6411bde7-2df7-4716-af46-938a3a8ce0bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9\" (UID: \"6411bde7-2df7-4716-af46-938a3a8ce0bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"
Nov 03 20:15:21.427461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.426352    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kldb\" (UniqueName: \"kubernetes.io/projected/6411bde7-2df7-4716-af46-938a3a8ce0bc-kube-api-access-8kldb\") pod \"maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9\" (UID: \"6411bde7-2df7-4716-af46-938a3a8ce0bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"
Nov 03 20:15:21.444460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.440495    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnjqn\" (UniqueName: \"kubernetes.io/projected/36123290-c071-43cd-8731-a44a4d12597b-kube-api-access-gnjqn\") pod \"maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz\" (UID: \"36123290-c071-43cd-8731-a44a4d12597b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"
Nov 03 20:15:21.444460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.443733    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k" event={"ID":"ce12762c-ce10-424d-baed-f43ab92bef20","Type":"ContainerStarted","Data":"1c6e46b3db6e946ba373512cc4554b8715771eaa7ef880371ac4ccd1af61df84"}
Nov 03 20:15:21.451141 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.451104    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk" event={"ID":"5835ea77-61e4-452e-8240-ff5ac2d55242","Type":"ContainerDied","Data":"f1b8b515c076c91409e9a825fc591b23782679802fae66c461b375c53b7515d7"}
Nov 03 20:15:21.451451 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.451401    5600 scope.go:122] "RemoveContainer" containerID="7adf23cd89e75c02abc72e1ff8e365f67b5d1ae1ea47ed71c95d893fbe5734c6"
Nov 03 20:15:21.456478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.451686    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"
Nov 03 20:15:21.494887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.494857    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"
Nov 03 20:15:21.501282 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.500213    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj" event={"ID":"a1a7697e-c056-445c-93ca-1d1346a48d35","Type":"ContainerDied","Data":"96483d8a77a7022dbee0e6060afdad9a59988874bd0837f4a0d49b4b70e0d965"}
Nov 03 20:15:21.501426 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.500294    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"
Nov 03 20:15:21.527149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.509200    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"
Nov 03 20:15:21.586124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.584862    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"]
Nov 03 20:15:21.590703 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.589217    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn" event={"ID":"c6eebfb3-3269-4ae2-b530-6d5680c6b3d2","Type":"ContainerDied","Data":"d47f39d510602b624df9cbc1c0cd1e5302b7af99ed2674dfd5556911f39c1743"}
Nov 03 20:15:21.590703 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.589331    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"
Nov 03 20:15:21.592602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.592574    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-ee2323-29895615-mkllj"]
Nov 03 20:15:21.644726 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.644691    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a7697e-c056-445c-93ca-1d1346a48d35" path="/var/lib/kubelet/pods/a1a7697e-c056-445c-93ca-1d1346a48d35/volumes"
Nov 03 20:15:21.752336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.741993    5600 scope.go:122] "RemoveContainer" containerID="3a4cd50491e36d54fa7d4dc72b7a72a71f65056864cac1051b986aa44bf61495"
Nov 03 20:15:21.853582 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.853505    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"]
Nov 03 20:15:21.864522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.864484    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-120fe3-29895615-kngsk"]
Nov 03 20:15:21.875607 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.875545    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"]
Nov 03 20:15:21.876035 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.875914    5600 scope.go:122] "RemoveContainer" containerID="76aa34276dfd7240dc75799446e89404ccd96bf7d2f8099c1febb4dc7bddbc96"
Nov 03 20:15:21.892422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:21.892386    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-11a5b0-29895615-kttdn"]
Nov 03 20:15:22.035685 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:22.035493    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"]
Nov 03 20:15:22.066933 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:22.066907    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"]
Nov 03 20:15:22.069175 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:22.069092    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10bb390_2891_4be6_9550_f5dbfff9a352.slice/crio-06f119f38f74f8eba851ce25ecd8af704f22c23bfeaffde4ad2fe7c1c204e2e4 WatchSource:0}: Error finding container 06f119f38f74f8eba851ce25ecd8af704f22c23bfeaffde4ad2fe7c1c204e2e4: Status 404 returned error can't find the container with id 06f119f38f74f8eba851ce25ecd8af704f22c23bfeaffde4ad2fe7c1c204e2e4
Nov 03 20:15:22.142202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:22.142166    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"]
Nov 03 20:15:22.150141 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:22.150112    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36123290_c071_43cd_8731_a44a4d12597b.slice/crio-e85d067c7385eb2a7b02e506d37d8f362b0a15f7946a23f6f981f95c24c9dead WatchSource:0}: Error finding container e85d067c7385eb2a7b02e506d37d8f362b0a15f7946a23f6f981f95c24c9dead: Status 404 returned error can't find the container with id e85d067c7385eb2a7b02e506d37d8f362b0a15f7946a23f6f981f95c24c9dead
Nov 03 20:15:22.601953 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:22.601926    5600 generic.go:356] "Generic (PLEG): container finished" podID="ce12762c-ce10-424d-baed-f43ab92bef20" containerID="1c6e46b3db6e946ba373512cc4554b8715771eaa7ef880371ac4ccd1af61df84" exitCode=60
Nov 03 20:15:22.602387 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:22.602084    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k" event={"ID":"ce12762c-ce10-424d-baed-f43ab92bef20","Type":"ContainerDied","Data":"1c6e46b3db6e946ba373512cc4554b8715771eaa7ef880371ac4ccd1af61df84"}
Nov 03 20:15:22.602387 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:22.602142    5600 scope.go:122] "RemoveContainer" containerID="2691fbd292eebe0807b31defd9f700fcc0eab8d3987c4ba004d14d63ad5814b7"
Nov 03 20:15:22.602717 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:22.602702    5600 scope.go:122] "RemoveContainer" containerID="1c6e46b3db6e946ba373512cc4554b8715771eaa7ef880371ac4ccd1af61df84"
Nov 03 20:15:22.603117 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:22.603093    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k_odh-ai-gateway-infra(ce12762c-ce10-424d-baed-f43ab92bef20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k" podUID="ce12762c-ce10-424d-baed-f43ab92bef20"
Nov 03 20:15:22.605616 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:22.605587    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz" event={"ID":"36123290-c071-43cd-8731-a44a4d12597b","Type":"ContainerStarted","Data":"e85d067c7385eb2a7b02e506d37d8f362b0a15f7946a23f6f981f95c24c9dead"}
Nov 03 20:15:22.607752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:22.607727    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs" event={"ID":"a10bb390-2891-4be6-9550-f5dbfff9a352","Type":"ContainerStarted","Data":"06f119f38f74f8eba851ce25ecd8af704f22c23bfeaffde4ad2fe7c1c204e2e4"}
Nov 03 20:15:22.612901 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:22.612876    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9" event={"ID":"6411bde7-2df7-4716-af46-938a3a8ce0bc","Type":"ContainerStarted","Data":"8ba961b417f23cda50e29581297be8f607701a22ee2d32c76a9559ef8a28a99d"}
Nov 03 20:15:23.679067 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.676102    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5835ea77-61e4-452e-8240-ff5ac2d55242" path="/var/lib/kubelet/pods/5835ea77-61e4-452e-8240-ff5ac2d55242/volumes"
Nov 03 20:15:23.681006 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.680977    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6eebfb3-3269-4ae2-b530-6d5680c6b3d2" path="/var/lib/kubelet/pods/c6eebfb3-3269-4ae2-b530-6d5680c6b3d2/volumes"
Nov 03 20:15:23.697141 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.697077    5600 scope.go:122] "RemoveContainer" containerID="1c6e46b3db6e946ba373512cc4554b8715771eaa7ef880371ac4ccd1af61df84"
Nov 03 20:15:23.698195 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:23.698140    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k_odh-ai-gateway-infra(ce12762c-ce10-424d-baed-f43ab92bef20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k" podUID="ce12762c-ce10-424d-baed-f43ab92bef20"
Nov 03 20:15:23.699656 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.699070    5600 generic.go:356] "Generic (PLEG): container finished" podID="36123290-c071-43cd-8731-a44a4d12597b" containerID="afe4057ae09d420eb247bcd80502e422e9759e5492f96d8ab0d21eeb476b069c" exitCode=60
Nov 03 20:15:23.699953 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.699093    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz" event={"ID":"36123290-c071-43cd-8731-a44a4d12597b","Type":"ContainerDied","Data":"afe4057ae09d420eb247bcd80502e422e9759e5492f96d8ab0d21eeb476b069c"}
Nov 03 20:15:23.722405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.699724    5600 scope.go:122] "RemoveContainer" containerID="afe4057ae09d420eb247bcd80502e422e9759e5492f96d8ab0d21eeb476b069c"
Nov 03 20:15:23.728598 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.728571    5600 generic.go:356] "Generic (PLEG): container finished" podID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerID="37ac5c9d44c04151a7edbea2d3f155e0824d485259023264bec09c06f20a4ee1" exitCode=60
Nov 03 20:15:23.731126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.731098    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs" event={"ID":"a10bb390-2891-4be6-9550-f5dbfff9a352","Type":"ContainerDied","Data":"37ac5c9d44c04151a7edbea2d3f155e0824d485259023264bec09c06f20a4ee1"}
Nov 03 20:15:23.737472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.734405    5600 scope.go:122] "RemoveContainer" containerID="37ac5c9d44c04151a7edbea2d3f155e0824d485259023264bec09c06f20a4ee1"
Nov 03 20:15:23.762118 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.762083    5600 generic.go:356] "Generic (PLEG): container finished" podID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerID="8094adf5ce814b1db663e0ab71d408f78134fb76daffbffbee83491865fb8d08" exitCode=60
Nov 03 20:15:23.762297 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.762148    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9" event={"ID":"6411bde7-2df7-4716-af46-938a3a8ce0bc","Type":"ContainerDied","Data":"8094adf5ce814b1db663e0ab71d408f78134fb76daffbffbee83491865fb8d08"}
Nov 03 20:15:23.762631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:23.762612    5600 scope.go:122] "RemoveContainer" containerID="8094adf5ce814b1db663e0ab71d408f78134fb76daffbffbee83491865fb8d08"
Nov 03 20:15:24.784559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:24.784459    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz" event={"ID":"36123290-c071-43cd-8731-a44a4d12597b","Type":"ContainerStarted","Data":"76a0b81473fbfece7eec4423c96fd7a5af92d3525cfaffb04379ccc8f13f5ef5"}
Nov 03 20:15:24.788345 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:24.788314    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs" event={"ID":"a10bb390-2891-4be6-9550-f5dbfff9a352","Type":"ContainerStarted","Data":"7ee0e4abe91a2556c31c73dcfc8877e8fb8c4bba583fdc13067894378b640daf"}
Nov 03 20:15:24.791166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:24.791138    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9" event={"ID":"6411bde7-2df7-4716-af46-938a3a8ce0bc","Type":"ContainerStarted","Data":"ee527de12cac4feeaf87fa0c616395553c4bfd89d066912e77226494ea101ee5"}
Nov 03 20:15:24.812318 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:24.812269    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz" podStartSLOduration=24.812250947 podStartE2EDuration="24.812250947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:24.810187757 +0000 UTC m=+2107.323009542" watchObservedRunningTime="2026-11-03 20:15:24.812250947 +0000 UTC m=+2107.325072729"
Nov 03 20:15:24.839322 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:24.839257    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs" podStartSLOduration=24.83923716 podStartE2EDuration="24.83923716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:24.832870451 +0000 UTC m=+2107.345692233" watchObservedRunningTime="2026-11-03 20:15:24.83923716 +0000 UTC m=+2107.352058943"
Nov 03 20:15:24.863590 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:24.862853    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9" podStartSLOduration=24.862831646 podStartE2EDuration="24.862831646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:24.854880116 +0000 UTC m=+2107.367701901" watchObservedRunningTime="2026-11-03 20:15:24.862831646 +0000 UTC m=+2107.375653429"
Nov 03 20:15:25.801928 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.801888    5600 generic.go:356] "Generic (PLEG): container finished" podID="36123290-c071-43cd-8731-a44a4d12597b" containerID="76a0b81473fbfece7eec4423c96fd7a5af92d3525cfaffb04379ccc8f13f5ef5" exitCode=60
Nov 03 20:15:25.802514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.801978    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz" event={"ID":"36123290-c071-43cd-8731-a44a4d12597b","Type":"ContainerDied","Data":"76a0b81473fbfece7eec4423c96fd7a5af92d3525cfaffb04379ccc8f13f5ef5"}
Nov 03 20:15:25.802514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.802047    5600 scope.go:122] "RemoveContainer" containerID="afe4057ae09d420eb247bcd80502e422e9759e5492f96d8ab0d21eeb476b069c"
Nov 03 20:15:25.802514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.802333    5600 scope.go:122] "RemoveContainer" containerID="76a0b81473fbfece7eec4423c96fd7a5af92d3525cfaffb04379ccc8f13f5ef5"
Nov 03 20:15:25.802979 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:25.802938    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz_odh-ai-gateway-infra(36123290-c071-43cd-8731-a44a4d12597b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz" podUID="36123290-c071-43cd-8731-a44a4d12597b"
Nov 03 20:15:25.805145 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.805121    5600 generic.go:356] "Generic (PLEG): container finished" podID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerID="7ee0e4abe91a2556c31c73dcfc8877e8fb8c4bba583fdc13067894378b640daf" exitCode=60
Nov 03 20:15:25.805300 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.805239    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs" event={"ID":"a10bb390-2891-4be6-9550-f5dbfff9a352","Type":"ContainerDied","Data":"7ee0e4abe91a2556c31c73dcfc8877e8fb8c4bba583fdc13067894378b640daf"}
Nov 03 20:15:25.805614 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.805593    5600 scope.go:122] "RemoveContainer" containerID="7ee0e4abe91a2556c31c73dcfc8877e8fb8c4bba583fdc13067894378b640daf"
Nov 03 20:15:25.806023 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:25.805994    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs_odh-ai-gateway-infra(a10bb390-2891-4be6-9550-f5dbfff9a352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352"
Nov 03 20:15:25.807694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.807666    5600 generic.go:356] "Generic (PLEG): container finished" podID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerID="ee527de12cac4feeaf87fa0c616395553c4bfd89d066912e77226494ea101ee5" exitCode=60
Nov 03 20:15:25.807771 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.807751    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9" event={"ID":"6411bde7-2df7-4716-af46-938a3a8ce0bc","Type":"ContainerDied","Data":"ee527de12cac4feeaf87fa0c616395553c4bfd89d066912e77226494ea101ee5"}
Nov 03 20:15:25.808390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.808369    5600 scope.go:122] "RemoveContainer" containerID="ee527de12cac4feeaf87fa0c616395553c4bfd89d066912e77226494ea101ee5"
Nov 03 20:15:25.808920 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:25.808897    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9_odh-ai-gateway-infra(6411bde7-2df7-4716-af46-938a3a8ce0bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc"
Nov 03 20:15:25.820035 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.819666    5600 scope.go:122] "RemoveContainer" containerID="37ac5c9d44c04151a7edbea2d3f155e0824d485259023264bec09c06f20a4ee1"
Nov 03 20:15:25.838815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:25.838794    5600 scope.go:122] "RemoveContainer" containerID="8094adf5ce814b1db663e0ab71d408f78134fb76daffbffbee83491865fb8d08"
Nov 03 20:15:26.817952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:26.817913    5600 scope.go:122] "RemoveContainer" containerID="7ee0e4abe91a2556c31c73dcfc8877e8fb8c4bba583fdc13067894378b640daf"
Nov 03 20:15:26.818508 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:26.818420    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs_odh-ai-gateway-infra(a10bb390-2891-4be6-9550-f5dbfff9a352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352"
Nov 03 20:15:26.819992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:26.819969    5600 scope.go:122] "RemoveContainer" containerID="ee527de12cac4feeaf87fa0c616395553c4bfd89d066912e77226494ea101ee5"
Nov 03 20:15:26.820403 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:26.820379    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9_odh-ai-gateway-infra(6411bde7-2df7-4716-af46-938a3a8ce0bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc"
Nov 03 20:15:26.822244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:26.822225    5600 scope.go:122] "RemoveContainer" containerID="76a0b81473fbfece7eec4423c96fd7a5af92d3525cfaffb04379ccc8f13f5ef5"
Nov 03 20:15:26.822584 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:26.822564    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz_odh-ai-gateway-infra(36123290-c071-43cd-8731-a44a4d12597b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz" podUID="36123290-c071-43cd-8731-a44a4d12597b"
Nov 03 20:15:28.813489 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:28.811841    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:15:35.611868 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:35.611645    5600 scope.go:122] "RemoveContainer" containerID="ee527de12cac4feeaf87fa0c616395553c4bfd89d066912e77226494ea101ee5"
Nov 03 20:15:35.611868 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:35.611750    5600 scope.go:122] "RemoveContainer" containerID="1c6e46b3db6e946ba373512cc4554b8715771eaa7ef880371ac4ccd1af61df84"
Nov 03 20:15:36.609763 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:36.609733    5600 scope.go:122] "RemoveContainer" containerID="7ee0e4abe91a2556c31c73dcfc8877e8fb8c4bba583fdc13067894378b640daf"
Nov 03 20:15:36.923846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:36.923784    5600 generic.go:356] "Generic (PLEG): container finished" podID="ce12762c-ce10-424d-baed-f43ab92bef20" containerID="36a19da5588394d08d4b4549a9e554e44b792013d4464f2b8c2da5c27a1fd187" exitCode=60
Nov 03 20:15:36.924648 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:36.923884    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k" event={"ID":"ce12762c-ce10-424d-baed-f43ab92bef20","Type":"ContainerDied","Data":"36a19da5588394d08d4b4549a9e554e44b792013d4464f2b8c2da5c27a1fd187"}
Nov 03 20:15:36.924648 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:36.923973    5600 scope.go:122] "RemoveContainer" containerID="1c6e46b3db6e946ba373512cc4554b8715771eaa7ef880371ac4ccd1af61df84"
Nov 03 20:15:36.924648 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:36.924547    5600 scope.go:122] "RemoveContainer" containerID="36a19da5588394d08d4b4549a9e554e44b792013d4464f2b8c2da5c27a1fd187"
Nov 03 20:15:36.925022 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:36.924991    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k_odh-ai-gateway-infra(ce12762c-ce10-424d-baed-f43ab92bef20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k" podUID="ce12762c-ce10-424d-baed-f43ab92bef20"
Nov 03 20:15:36.926848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:36.926826    5600 generic.go:356] "Generic (PLEG): container finished" podID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerID="92bb35e75596b385aafeff5be40ba0f1dfc822e063f60fe1e991aebee2329998" exitCode=60
Nov 03 20:15:36.926976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:36.926871    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9" event={"ID":"6411bde7-2df7-4716-af46-938a3a8ce0bc","Type":"ContainerDied","Data":"92bb35e75596b385aafeff5be40ba0f1dfc822e063f60fe1e991aebee2329998"}
Nov 03 20:15:36.927272 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:36.927248    5600 scope.go:122] "RemoveContainer" containerID="92bb35e75596b385aafeff5be40ba0f1dfc822e063f60fe1e991aebee2329998"
Nov 03 20:15:36.927639 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:36.927612    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9_odh-ai-gateway-infra(6411bde7-2df7-4716-af46-938a3a8ce0bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc"
Nov 03 20:15:36.942244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:36.942221    5600 scope.go:122] "RemoveContainer" containerID="ee527de12cac4feeaf87fa0c616395553c4bfd89d066912e77226494ea101ee5"
Nov 03 20:15:37.619970 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:37.619941    5600 scope.go:122] "RemoveContainer" containerID="76a0b81473fbfece7eec4423c96fd7a5af92d3525cfaffb04379ccc8f13f5ef5"
Nov 03 20:15:37.940261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:37.940223    5600 generic.go:356] "Generic (PLEG): container finished" podID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerID="d1e80982155f4c277b436d39d929905e42d64283114febb973db03f1fe051029" exitCode=60
Nov 03 20:15:37.940741 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:37.940296    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs" event={"ID":"a10bb390-2891-4be6-9550-f5dbfff9a352","Type":"ContainerDied","Data":"d1e80982155f4c277b436d39d929905e42d64283114febb973db03f1fe051029"}
Nov 03 20:15:37.940741 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:37.940352    5600 scope.go:122] "RemoveContainer" containerID="7ee0e4abe91a2556c31c73dcfc8877e8fb8c4bba583fdc13067894378b640daf"
Nov 03 20:15:37.940865 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:37.940832    5600 scope.go:122] "RemoveContainer" containerID="d1e80982155f4c277b436d39d929905e42d64283114febb973db03f1fe051029"
Nov 03 20:15:37.941239 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:37.941215    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs_odh-ai-gateway-infra(a10bb390-2891-4be6-9550-f5dbfff9a352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352"
Nov 03 20:15:37.954504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:37.954473    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"]
Nov 03 20:15:37.972058 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:37.972035    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"]
Nov 03 20:15:38.168527 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.168451    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"
Nov 03 20:15:38.218199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.218152    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce12762c-ce10-424d-baed-f43ab92bef20-kube-api-access-wfjx7\" (UniqueName: \"kubernetes.io/projected/ce12762c-ce10-424d-baed-f43ab92bef20-kube-api-access-wfjx7\") pod \"ce12762c-ce10-424d-baed-f43ab92bef20\" (UID: \"ce12762c-ce10-424d-baed-f43ab92bef20\") "
Nov 03 20:15:38.218377 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.218262    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce12762c-ce10-424d-baed-f43ab92bef20-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce12762c-ce10-424d-baed-f43ab92bef20-maas-api-service-ca\") pod \"ce12762c-ce10-424d-baed-f43ab92bef20\" (UID: \"ce12762c-ce10-424d-baed-f43ab92bef20\") "
Nov 03 20:15:38.219800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.219770    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce12762c-ce10-424d-baed-f43ab92bef20-maas-api-service-ca" pod "ce12762c-ce10-424d-baed-f43ab92bef20" (UID: "ce12762c-ce10-424d-baed-f43ab92bef20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:38.222395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.222350    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce12762c-ce10-424d-baed-f43ab92bef20-kube-api-access-wfjx7" pod "ce12762c-ce10-424d-baed-f43ab92bef20" (UID: "ce12762c-ce10-424d-baed-f43ab92bef20"). InnerVolumeSpecName "kube-api-access-wfjx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:38.244847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.244823    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"
Nov 03 20:15:38.274668 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.274635    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"]
Nov 03 20:15:38.275355 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275328    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ce12762c-ce10-424d-baed-f43ab92bef20" containerName="cleanup"
Nov 03 20:15:38.275355 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275345    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce12762c-ce10-424d-baed-f43ab92bef20" containerName="cleanup"
Nov 03 20:15:38.275355 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275355    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ce12762c-ce10-424d-baed-f43ab92bef20" containerName="cleanup"
Nov 03 20:15:38.275355 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275360    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce12762c-ce10-424d-baed-f43ab92bef20" containerName="cleanup"
Nov 03 20:15:38.275691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275371    5600 cpu_manager.go:414] "containerMap: removing container" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerName="cleanup"
Nov 03 20:15:38.275691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275376    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerName="cleanup"
Nov 03 20:15:38.275691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275394    5600 cpu_manager.go:414] "containerMap: removing container" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerName="cleanup"
Nov 03 20:15:38.275691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275399    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerName="cleanup"
Nov 03 20:15:38.275691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275549    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce12762c-ce10-424d-baed-f43ab92bef20" containerName="cleanup"
Nov 03 20:15:38.275691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275558    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerName="cleanup"
Nov 03 20:15:38.275691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275567    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerName="cleanup"
Nov 03 20:15:38.275691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.275582    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce12762c-ce10-424d-baed-f43ab92bef20" containerName="cleanup"
Nov 03 20:15:38.280831 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.280035    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"
Nov 03 20:15:38.313738 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.308842    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"]
Nov 03 20:15:38.315142 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.315117    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"]
Nov 03 20:15:38.316055 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.316032    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ce12762c-ce10-424d-baed-f43ab92bef20" containerName="cleanup"
Nov 03 20:15:38.316055 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.316054    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce12762c-ce10-424d-baed-f43ab92bef20" containerName="cleanup"
Nov 03 20:15:38.316223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.316064    5600 cpu_manager.go:414] "containerMap: removing container" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerName="cleanup"
Nov 03 20:15:38.316223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.316071    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerName="cleanup"
Nov 03 20:15:38.316223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.316161    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce12762c-ce10-424d-baed-f43ab92bef20" containerName="cleanup"
Nov 03 20:15:38.316223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.316193    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc" containerName="cleanup"
Nov 03 20:15:38.320258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.320233    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6411bde7-2df7-4716-af46-938a3a8ce0bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6411bde7-2df7-4716-af46-938a3a8ce0bc-maas-api-service-ca\") pod \"6411bde7-2df7-4716-af46-938a3a8ce0bc\" (UID: \"6411bde7-2df7-4716-af46-938a3a8ce0bc\") "
Nov 03 20:15:38.320370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.320328    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6411bde7-2df7-4716-af46-938a3a8ce0bc-kube-api-access-8kldb\" (UniqueName: \"kubernetes.io/projected/6411bde7-2df7-4716-af46-938a3a8ce0bc-kube-api-access-8kldb\") pod \"6411bde7-2df7-4716-af46-938a3a8ce0bc\" (UID: \"6411bde7-2df7-4716-af46-938a3a8ce0bc\") "
Nov 03 20:15:38.320572 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.320554    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"
Nov 03 20:15:38.320908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.320616    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6411bde7-2df7-4716-af46-938a3a8ce0bc-maas-api-service-ca" pod "6411bde7-2df7-4716-af46-938a3a8ce0bc" (UID: "6411bde7-2df7-4716-af46-938a3a8ce0bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:38.321885 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.321820    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6411bde7-2df7-4716-af46-938a3a8ce0bc-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:38.321885 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.321847    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfjx7\" (UniqueName: \"kubernetes.io/projected/ce12762c-ce10-424d-baed-f43ab92bef20-kube-api-access-wfjx7\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:38.321885 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.321863    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce12762c-ce10-424d-baed-f43ab92bef20-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:38.344480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.344421    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"]
Nov 03 20:15:38.382279 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.382210    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6411bde7-2df7-4716-af46-938a3a8ce0bc-kube-api-access-8kldb" pod "6411bde7-2df7-4716-af46-938a3a8ce0bc" (UID: "6411bde7-2df7-4716-af46-938a3a8ce0bc"). InnerVolumeSpecName "kube-api-access-8kldb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:38.423667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.423566    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w75g\" (UniqueName: \"kubernetes.io/projected/ba818408-2dbb-489d-b101-ca955de8aedf-kube-api-access-5w75g\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp\" (UID: \"ba818408-2dbb-489d-b101-ca955de8aedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"
Nov 03 20:15:38.423822 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.423730    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwshm\" (UniqueName: \"kubernetes.io/projected/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-kube-api-access-wwshm\") pod \"maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk\" (UID: \"3c35eed5-b7e1-4792-838e-cfde3d6de8d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"
Nov 03 20:15:38.423976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.423951    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk\" (UID: \"3c35eed5-b7e1-4792-838e-cfde3d6de8d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"
Nov 03 20:15:38.424185 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.424167    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba818408-2dbb-489d-b101-ca955de8aedf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp\" (UID: \"ba818408-2dbb-489d-b101-ca955de8aedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"
Nov 03 20:15:38.424313 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.424300    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kldb\" (UniqueName: \"kubernetes.io/projected/6411bde7-2df7-4716-af46-938a3a8ce0bc-kube-api-access-8kldb\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:38.525328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.525292    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba818408-2dbb-489d-b101-ca955de8aedf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp\" (UID: \"ba818408-2dbb-489d-b101-ca955de8aedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"
Nov 03 20:15:38.525550 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.525370    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w75g\" (UniqueName: \"kubernetes.io/projected/ba818408-2dbb-489d-b101-ca955de8aedf-kube-api-access-5w75g\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp\" (UID: \"ba818408-2dbb-489d-b101-ca955de8aedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"
Nov 03 20:15:38.525550 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.525427    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwshm\" (UniqueName: \"kubernetes.io/projected/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-kube-api-access-wwshm\") pod \"maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk\" (UID: \"3c35eed5-b7e1-4792-838e-cfde3d6de8d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"
Nov 03 20:15:38.525683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.525598    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk\" (UID: \"3c35eed5-b7e1-4792-838e-cfde3d6de8d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"
Nov 03 20:15:38.526219 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.526193    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba818408-2dbb-489d-b101-ca955de8aedf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp\" (UID: \"ba818408-2dbb-489d-b101-ca955de8aedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"
Nov 03 20:15:38.526292 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.526208    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk\" (UID: \"3c35eed5-b7e1-4792-838e-cfde3d6de8d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"
Nov 03 20:15:38.539405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.539381    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w75g\" (UniqueName: \"kubernetes.io/projected/ba818408-2dbb-489d-b101-ca955de8aedf-kube-api-access-5w75g\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp\" (UID: \"ba818408-2dbb-489d-b101-ca955de8aedf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"
Nov 03 20:15:38.539567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.539486    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwshm\" (UniqueName: \"kubernetes.io/projected/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-kube-api-access-wwshm\") pod \"maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk\" (UID: \"3c35eed5-b7e1-4792-838e-cfde3d6de8d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"
Nov 03 20:15:38.601651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.601609    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"
Nov 03 20:15:38.643644 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.643609    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"
Nov 03 20:15:38.888183 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:38.888151    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba818408_2dbb_489d_b101_ca955de8aedf.slice/crio-91deafac96d8a7e3b775802ef37ac4c3696759df232a8427727b36ccfb40d9c5 WatchSource:0}: Error finding container 91deafac96d8a7e3b775802ef37ac4c3696759df232a8427727b36ccfb40d9c5: Status 404 returned error can't find the container with id 91deafac96d8a7e3b775802ef37ac4c3696759df232a8427727b36ccfb40d9c5
Nov 03 20:15:38.889352 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.889319    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"]
Nov 03 20:15:38.961121 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.961045    5600 generic.go:356] "Generic (PLEG): container finished" podID="36123290-c071-43cd-8731-a44a4d12597b" containerID="6baa596aa178a5f621f9237953bea2838d3e1771ee7979f0e83461b3ebab177a" exitCode=60
Nov 03 20:15:38.961597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.961160    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz" event={"ID":"36123290-c071-43cd-8731-a44a4d12597b","Type":"ContainerDied","Data":"6baa596aa178a5f621f9237953bea2838d3e1771ee7979f0e83461b3ebab177a"}
Nov 03 20:15:38.961597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.961383    5600 scope.go:122] "RemoveContainer" containerID="76a0b81473fbfece7eec4423c96fd7a5af92d3525cfaffb04379ccc8f13f5ef5"
Nov 03 20:15:38.961888 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.961860    5600 scope.go:122] "RemoveContainer" containerID="6baa596aa178a5f621f9237953bea2838d3e1771ee7979f0e83461b3ebab177a"
Nov 03 20:15:38.962334 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:38.962272    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz_odh-ai-gateway-infra(36123290-c071-43cd-8731-a44a4d12597b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz" podUID="36123290-c071-43cd-8731-a44a4d12597b"
Nov 03 20:15:38.964851 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.964830    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"
Nov 03 20:15:38.964851 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.964840    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9" event={"ID":"6411bde7-2df7-4716-af46-938a3a8ce0bc","Type":"ContainerDied","Data":"8ba961b417f23cda50e29581297be8f607701a22ee2d32c76a9559ef8a28a99d"}
Nov 03 20:15:38.966722 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.966693    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" event={"ID":"ba818408-2dbb-489d-b101-ca955de8aedf","Type":"ContainerStarted","Data":"91deafac96d8a7e3b775802ef37ac4c3696759df232a8427727b36ccfb40d9c5"}
Nov 03 20:15:38.973739 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.973548    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k" event={"ID":"ce12762c-ce10-424d-baed-f43ab92bef20","Type":"ContainerDied","Data":"24901a32e5db03af7fb642b3129a25e445a542e54020fcc1f45fa257e6195baf"}
Nov 03 20:15:38.973739 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.973698    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"
Nov 03 20:15:38.977412 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.977392    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"]
Nov 03 20:15:38.986349 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.985715    5600 scope.go:122] "RemoveContainer" containerID="92bb35e75596b385aafeff5be40ba0f1dfc822e063f60fe1e991aebee2329998"
Nov 03 20:15:38.988085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:38.988021    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"]
Nov 03 20:15:39.019199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.019168    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"]
Nov 03 20:15:39.027762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.027724    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2092ba-29895615-sx5d9"]
Nov 03 20:15:39.039477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.039447    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"]
Nov 03 20:15:39.044752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.044730    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ab1e46-29895615-8zk8k"]
Nov 03 20:15:39.091471 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.091418    5600 scope.go:122] "RemoveContainer" containerID="36a19da5588394d08d4b4549a9e554e44b792013d4464f2b8c2da5c27a1fd187"
Nov 03 20:15:39.337860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.337836    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"
Nov 03 20:15:39.372355 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.372324    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"]
Nov 03 20:15:39.373108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.373089    5600 cpu_manager.go:414] "containerMap: removing container" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerName="cleanup"
Nov 03 20:15:39.373108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.373107    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerName="cleanup"
Nov 03 20:15:39.373339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.373125    5600 cpu_manager.go:414] "containerMap: removing container" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerName="cleanup"
Nov 03 20:15:39.373339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.373130    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerName="cleanup"
Nov 03 20:15:39.373339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.373290    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerName="cleanup"
Nov 03 20:15:39.373339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.373303    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerName="cleanup"
Nov 03 20:15:39.373339 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.373312    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerName="cleanup"
Nov 03 20:15:39.378986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.378474    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"
Nov 03 20:15:39.380917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.380896    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"]
Nov 03 20:15:39.440399 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.440360    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10bb390-2891-4be6-9550-f5dbfff9a352-kube-api-access-pvq7k\" (UniqueName: \"kubernetes.io/projected/a10bb390-2891-4be6-9550-f5dbfff9a352-kube-api-access-pvq7k\") pod \"a10bb390-2891-4be6-9550-f5dbfff9a352\" (UID: \"a10bb390-2891-4be6-9550-f5dbfff9a352\") "
Nov 03 20:15:39.440618 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.440571    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10bb390-2891-4be6-9550-f5dbfff9a352-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10bb390-2891-4be6-9550-f5dbfff9a352-maas-api-service-ca\") pod \"a10bb390-2891-4be6-9550-f5dbfff9a352\" (UID: \"a10bb390-2891-4be6-9550-f5dbfff9a352\") "
Nov 03 20:15:39.440929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.440902    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10bb390-2891-4be6-9550-f5dbfff9a352-maas-api-service-ca" pod "a10bb390-2891-4be6-9550-f5dbfff9a352" (UID: "a10bb390-2891-4be6-9550-f5dbfff9a352"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:39.441039 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.441018    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97\" (UID: \"c8c6716a-c78a-4aa8-9ead-eba203f65cc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"
Nov 03 20:15:39.441212 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.441194    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzk5n\" (UniqueName: \"kubernetes.io/projected/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-kube-api-access-rzk5n\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97\" (UID: \"c8c6716a-c78a-4aa8-9ead-eba203f65cc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"
Nov 03 20:15:39.441519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.441502    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10bb390-2891-4be6-9550-f5dbfff9a352-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:39.442833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.442805    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10bb390-2891-4be6-9550-f5dbfff9a352-kube-api-access-pvq7k" pod "a10bb390-2891-4be6-9550-f5dbfff9a352" (UID: "a10bb390-2891-4be6-9550-f5dbfff9a352"). InnerVolumeSpecName "kube-api-access-pvq7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:39.543335 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.543253    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97\" (UID: \"c8c6716a-c78a-4aa8-9ead-eba203f65cc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"
Nov 03 20:15:39.543509 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.543398    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzk5n\" (UniqueName: \"kubernetes.io/projected/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-kube-api-access-rzk5n\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97\" (UID: \"c8c6716a-c78a-4aa8-9ead-eba203f65cc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"
Nov 03 20:15:39.543509 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.543487    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvq7k\" (UniqueName: \"kubernetes.io/projected/a10bb390-2891-4be6-9550-f5dbfff9a352-kube-api-access-pvq7k\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:39.543985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.543962    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97\" (UID: \"c8c6716a-c78a-4aa8-9ead-eba203f65cc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"
Nov 03 20:15:39.551710 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.551687    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzk5n\" (UniqueName: \"kubernetes.io/projected/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-kube-api-access-rzk5n\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97\" (UID: \"c8c6716a-c78a-4aa8-9ead-eba203f65cc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"
Nov 03 20:15:39.618084 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.618047    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6411bde7-2df7-4716-af46-938a3a8ce0bc" path="/var/lib/kubelet/pods/6411bde7-2df7-4716-af46-938a3a8ce0bc/volumes"
Nov 03 20:15:39.618679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.618660    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce12762c-ce10-424d-baed-f43ab92bef20" path="/var/lib/kubelet/pods/ce12762c-ce10-424d-baed-f43ab92bef20/volumes"
Nov 03 20:15:39.709759 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.709722    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"
Nov 03 20:15:39.988340 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.988303    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs" event={"ID":"a10bb390-2891-4be6-9550-f5dbfff9a352","Type":"ContainerDied","Data":"06f119f38f74f8eba851ce25ecd8af704f22c23bfeaffde4ad2fe7c1c204e2e4"}
Nov 03 20:15:39.988800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.988343    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"
Nov 03 20:15:39.988800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.988384    5600 scope.go:122] "RemoveContainer" containerID="d1e80982155f4c277b436d39d929905e42d64283114febb973db03f1fe051029"
Nov 03 20:15:39.992045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.992015    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" event={"ID":"ba818408-2dbb-489d-b101-ca955de8aedf","Type":"ContainerStarted","Data":"0692ae06c281d6aa54f92434c8d32b05d5ed2680a0a1f894f00fc08470eb5a75"}
Nov 03 20:15:39.996246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.996216    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" event={"ID":"3c35eed5-b7e1-4792-838e-cfde3d6de8d9","Type":"ContainerStarted","Data":"8be3adc5964362831af55021cb014521632e2624e3dbe179b50dd3d22a7abee4"}
Nov 03 20:15:39.996389 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:39.996270    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" event={"ID":"3c35eed5-b7e1-4792-838e-cfde3d6de8d9","Type":"ContainerStarted","Data":"b4d8434cc9688786492b4f4f6f6cde4e73ac56f46bfd374bb158731a144d140f"}
Nov 03 20:15:40.006476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.006450    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"]
Nov 03 20:15:40.008467 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:40.008415    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c6716a_c78a_4aa8_9ead_eba203f65cc1.slice/crio-749a0fdbc50f363f07965fbd32c0878145da9409e617c30f38b347a3c84ece0f WatchSource:0}: Error finding container 749a0fdbc50f363f07965fbd32c0878145da9409e617c30f38b347a3c84ece0f: Status 404 returned error can't find the container with id 749a0fdbc50f363f07965fbd32c0878145da9409e617c30f38b347a3c84ece0f
Nov 03 20:15:40.014972 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.014946    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"]
Nov 03 20:15:40.037022 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.036983    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"]
Nov 03 20:15:40.050962 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.050932    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b42aea-29895615-2v7bs"]
Nov 03 20:15:40.053540 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.051975    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" podStartSLOduration=40.051958385 podStartE2EDuration="40.051958385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:40.034538408 +0000 UTC m=+2122.547360193" watchObservedRunningTime="2026-11-03 20:15:40.051958385 +0000 UTC m=+2122.564780168"
Nov 03 20:15:40.058368 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.058327    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" podStartSLOduration=40.058315419 podStartE2EDuration="40.058315419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:40.050321655 +0000 UTC m=+2122.563143439" watchObservedRunningTime="2026-11-03 20:15:40.058315419 +0000 UTC m=+2122.571137198"
Nov 03 20:15:40.275791 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.275727    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"
Nov 03 20:15:40.307655 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.307621    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"]
Nov 03 20:15:40.308747 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.308718    5600 cpu_manager.go:414] "containerMap: removing container" podUID="36123290-c071-43cd-8731-a44a4d12597b" containerName="cleanup"
Nov 03 20:15:40.308747 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.308746    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36123290-c071-43cd-8731-a44a4d12597b" containerName="cleanup"
Nov 03 20:15:40.308919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.308788    5600 cpu_manager.go:414] "containerMap: removing container" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerName="cleanup"
Nov 03 20:15:40.308919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.308797    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352" containerName="cleanup"
Nov 03 20:15:40.308919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.308872    5600 cpu_manager.go:414] "containerMap: removing container" podUID="36123290-c071-43cd-8731-a44a4d12597b" containerName="cleanup"
Nov 03 20:15:40.308919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.308879    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36123290-c071-43cd-8731-a44a4d12597b" containerName="cleanup"
Nov 03 20:15:40.309046 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.309001    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="36123290-c071-43cd-8731-a44a4d12597b" containerName="cleanup"
Nov 03 20:15:40.309046 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.309012    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="36123290-c071-43cd-8731-a44a4d12597b" containerName="cleanup"
Nov 03 20:15:40.319171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.315210    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:40.320686 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.320662    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:15:40.323659 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.323580    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"]
Nov 03 20:15:40.357753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.357712    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36123290-c071-43cd-8731-a44a4d12597b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36123290-c071-43cd-8731-a44a4d12597b-maas-api-service-ca\") pod \"36123290-c071-43cd-8731-a44a4d12597b\" (UID: \"36123290-c071-43cd-8731-a44a4d12597b\") "
Nov 03 20:15:40.358084 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.358056    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36123290-c071-43cd-8731-a44a4d12597b-maas-api-service-ca" pod "36123290-c071-43cd-8731-a44a4d12597b" (UID: "36123290-c071-43cd-8731-a44a4d12597b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:40.358205 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.358137    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36123290-c071-43cd-8731-a44a4d12597b-kube-api-access-gnjqn\" (UniqueName: \"kubernetes.io/projected/36123290-c071-43cd-8731-a44a4d12597b-kube-api-access-gnjqn\") pod \"36123290-c071-43cd-8731-a44a4d12597b\" (UID: \"36123290-c071-43cd-8731-a44a4d12597b\") "
Nov 03 20:15:40.358821 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.358788    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a865d6c2-f2tvc\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:40.359398 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.359163    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a865d6c2-f2tvc\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:40.359398 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.359219    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84m2d\" (UniqueName: \"kubernetes.io/projected/fd38fad9-7ea8-4f12-8b53-747ed99e901b-kube-api-access-84m2d\") pod \"maas-api-revoke-sub-a865d6c2-f2tvc\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:40.359398 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.359324    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36123290-c071-43cd-8731-a44a4d12597b-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:40.360918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.360888    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36123290-c071-43cd-8731-a44a4d12597b-kube-api-access-gnjqn" pod "36123290-c071-43cd-8731-a44a4d12597b" (UID: "36123290-c071-43cd-8731-a44a4d12597b"). InnerVolumeSpecName "kube-api-access-gnjqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:40.460830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.460794    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a865d6c2-f2tvc\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:40.461038 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.460906    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a865d6c2-f2tvc\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:40.461038 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.460949    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84m2d\" (UniqueName: \"kubernetes.io/projected/fd38fad9-7ea8-4f12-8b53-747ed99e901b-kube-api-access-84m2d\") pod \"maas-api-revoke-sub-a865d6c2-f2tvc\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:40.461182 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.461164    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnjqn\" (UniqueName: \"kubernetes.io/projected/36123290-c071-43cd-8731-a44a4d12597b-kube-api-access-gnjqn\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:40.461592 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.461571    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a865d6c2-f2tvc\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:40.463475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.463459    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a865d6c2-f2tvc\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:40.468825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.468808    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84m2d\" (UniqueName: \"kubernetes.io/projected/fd38fad9-7ea8-4f12-8b53-747ed99e901b-kube-api-access-84m2d\") pod \"maas-api-revoke-sub-a865d6c2-f2tvc\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:40.642120 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.642082    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:40.972514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:40.972487    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"]
Nov 03 20:15:41.029553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.026402    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc" event={"ID":"fd38fad9-7ea8-4f12-8b53-747ed99e901b","Type":"ContainerStarted","Data":"62366fee57f630c2553c5254c855f54abba7daa4bff9987c7081dfd43c629314"}
Nov 03 20:15:41.029553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.028360    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz" event={"ID":"36123290-c071-43cd-8731-a44a4d12597b","Type":"ContainerDied","Data":"e85d067c7385eb2a7b02e506d37d8f362b0a15f7946a23f6f981f95c24c9dead"}
Nov 03 20:15:41.029553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.028420    5600 scope.go:122] "RemoveContainer" containerID="6baa596aa178a5f621f9237953bea2838d3e1771ee7979f0e83461b3ebab177a"
Nov 03 20:15:41.029553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.028599    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"
Nov 03 20:15:41.033931 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.033059    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" event={"ID":"c8c6716a-c78a-4aa8-9ead-eba203f65cc1","Type":"ContainerStarted","Data":"2b92515a6b9edf3509385061e1114fac3bda2c4f2fecc016ec7b390ec7da433d"}
Nov 03 20:15:41.033931 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.033109    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" event={"ID":"c8c6716a-c78a-4aa8-9ead-eba203f65cc1","Type":"ContainerStarted","Data":"749a0fdbc50f363f07965fbd32c0878145da9409e617c30f38b347a3c84ece0f"}
Nov 03 20:15:41.037519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.037383    5600 generic.go:356] "Generic (PLEG): container finished" podID="ba818408-2dbb-489d-b101-ca955de8aedf" containerID="0692ae06c281d6aa54f92434c8d32b05d5ed2680a0a1f894f00fc08470eb5a75" exitCode=60
Nov 03 20:15:41.037519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.037501    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" event={"ID":"ba818408-2dbb-489d-b101-ca955de8aedf","Type":"ContainerDied","Data":"0692ae06c281d6aa54f92434c8d32b05d5ed2680a0a1f894f00fc08470eb5a75"}
Nov 03 20:15:41.037981 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.037923    5600 scope.go:122] "RemoveContainer" containerID="0692ae06c281d6aa54f92434c8d32b05d5ed2680a0a1f894f00fc08470eb5a75"
Nov 03 20:15:41.042907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.040504    5600 generic.go:356] "Generic (PLEG): container finished" podID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerID="8be3adc5964362831af55021cb014521632e2624e3dbe179b50dd3d22a7abee4" exitCode=60
Nov 03 20:15:41.042907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.040546    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" event={"ID":"3c35eed5-b7e1-4792-838e-cfde3d6de8d9","Type":"ContainerDied","Data":"8be3adc5964362831af55021cb014521632e2624e3dbe179b50dd3d22a7abee4"}
Nov 03 20:15:41.042907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.040907    5600 scope.go:122] "RemoveContainer" containerID="8be3adc5964362831af55021cb014521632e2624e3dbe179b50dd3d22a7abee4"
Nov 03 20:15:41.055940 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.055886    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" podStartSLOduration=41.055868546 podStartE2EDuration="41.055868546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:41.052607844 +0000 UTC m=+2123.565429627" watchObservedRunningTime="2026-11-03 20:15:41.055868546 +0000 UTC m=+2123.568690331"
Nov 03 20:15:41.095025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.094954    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"]
Nov 03 20:15:41.103134 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.103101    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b87733-29895615-2m4dz"]
Nov 03 20:15:41.625008 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.624972    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36123290-c071-43cd-8731-a44a4d12597b" path="/var/lib/kubelet/pods/36123290-c071-43cd-8731-a44a4d12597b/volumes"
Nov 03 20:15:41.625573 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:41.625546    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10bb390-2891-4be6-9550-f5dbfff9a352" path="/var/lib/kubelet/pods/a10bb390-2891-4be6-9550-f5dbfff9a352/volumes"
Nov 03 20:15:42.051280 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:42.051243    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" event={"ID":"ba818408-2dbb-489d-b101-ca955de8aedf","Type":"ContainerStarted","Data":"6e6bf238301f4c8efc3abae3ce79cbb3a421df8e9f2f44a6b6a8f1c7e39a8d8d"}
Nov 03 20:15:42.053276 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:42.053244    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" event={"ID":"3c35eed5-b7e1-4792-838e-cfde3d6de8d9","Type":"ContainerStarted","Data":"2c08a215b4c47d1c45332b43ef6b7370279304482487375eb6949ab3817c4802"}
Nov 03 20:15:42.055059 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:42.055032    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc" event={"ID":"fd38fad9-7ea8-4f12-8b53-747ed99e901b","Type":"ContainerStarted","Data":"49310ea5c57dffc52537c89cde5d4e80bb638e7323108623737b7646c662ac41"}
Nov 03 20:15:42.057705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:42.057684    5600 generic.go:356] "Generic (PLEG): container finished" podID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerID="2b92515a6b9edf3509385061e1114fac3bda2c4f2fecc016ec7b390ec7da433d" exitCode=60
Nov 03 20:15:42.057793 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:42.057733    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" event={"ID":"c8c6716a-c78a-4aa8-9ead-eba203f65cc1","Type":"ContainerDied","Data":"2b92515a6b9edf3509385061e1114fac3bda2c4f2fecc016ec7b390ec7da433d"}
Nov 03 20:15:42.057951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:42.057935    5600 scope.go:122] "RemoveContainer" containerID="2b92515a6b9edf3509385061e1114fac3bda2c4f2fecc016ec7b390ec7da433d"
Nov 03 20:15:42.114857 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:42.114649    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc" podStartSLOduration=39.114634857 podStartE2EDuration="39.114634857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:42.109363887 +0000 UTC m=+2124.622185681" watchObservedRunningTime="2026-11-03 20:15:42.114634857 +0000 UTC m=+2124.627456686"
Nov 03 20:15:43.070099 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:43.070064    5600 generic.go:356] "Generic (PLEG): container finished" podID="ba818408-2dbb-489d-b101-ca955de8aedf" containerID="6e6bf238301f4c8efc3abae3ce79cbb3a421df8e9f2f44a6b6a8f1c7e39a8d8d" exitCode=60
Nov 03 20:15:43.070688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:43.070151    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" event={"ID":"ba818408-2dbb-489d-b101-ca955de8aedf","Type":"ContainerDied","Data":"6e6bf238301f4c8efc3abae3ce79cbb3a421df8e9f2f44a6b6a8f1c7e39a8d8d"}
Nov 03 20:15:43.070688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:43.070280    5600 scope.go:122] "RemoveContainer" containerID="0692ae06c281d6aa54f92434c8d32b05d5ed2680a0a1f894f00fc08470eb5a75"
Nov 03 20:15:43.070688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:43.070579    5600 scope.go:122] "RemoveContainer" containerID="6e6bf238301f4c8efc3abae3ce79cbb3a421df8e9f2f44a6b6a8f1c7e39a8d8d"
Nov 03 20:15:43.071018 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:43.070993    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp_odh-ai-gateway-infra(ba818408-2dbb-489d-b101-ca955de8aedf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" podUID="ba818408-2dbb-489d-b101-ca955de8aedf"
Nov 03 20:15:43.073269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:43.073240    5600 generic.go:356] "Generic (PLEG): container finished" podID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerID="2c08a215b4c47d1c45332b43ef6b7370279304482487375eb6949ab3817c4802" exitCode=60
Nov 03 20:15:43.073386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:43.073320    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" event={"ID":"3c35eed5-b7e1-4792-838e-cfde3d6de8d9","Type":"ContainerDied","Data":"2c08a215b4c47d1c45332b43ef6b7370279304482487375eb6949ab3817c4802"}
Nov 03 20:15:43.073698 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:43.073673    5600 scope.go:122] "RemoveContainer" containerID="2c08a215b4c47d1c45332b43ef6b7370279304482487375eb6949ab3817c4802"
Nov 03 20:15:43.074128 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:43.074072    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk_odh-ai-gateway-infra(3c35eed5-b7e1-4792-838e-cfde3d6de8d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9"
Nov 03 20:15:43.076085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:43.076058    5600 generic.go:356] "Generic (PLEG): container finished" podID="fd38fad9-7ea8-4f12-8b53-747ed99e901b" containerID="49310ea5c57dffc52537c89cde5d4e80bb638e7323108623737b7646c662ac41" exitCode=0
Nov 03 20:15:43.076189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:43.076137    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc" event={"ID":"fd38fad9-7ea8-4f12-8b53-747ed99e901b","Type":"ContainerDied","Data":"49310ea5c57dffc52537c89cde5d4e80bb638e7323108623737b7646c662ac41"}
Nov 03 20:15:43.079359 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:43.079327    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" event={"ID":"c8c6716a-c78a-4aa8-9ead-eba203f65cc1","Type":"ContainerStarted","Data":"fef4ec10757c915dae968c54b3e4fe70e27fcacb301d328837b73f97f0def831"}
Nov 03 20:15:43.094495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:43.094451    5600 scope.go:122] "RemoveContainer" containerID="8be3adc5964362831af55021cb014521632e2624e3dbe179b50dd3d22a7abee4"
Nov 03 20:15:44.091862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.091820    5600 scope.go:122] "RemoveContainer" containerID="6e6bf238301f4c8efc3abae3ce79cbb3a421df8e9f2f44a6b6a8f1c7e39a8d8d"
Nov 03 20:15:44.092350 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:44.092276    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp_odh-ai-gateway-infra(ba818408-2dbb-489d-b101-ca955de8aedf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" podUID="ba818408-2dbb-489d-b101-ca955de8aedf"
Nov 03 20:15:44.094017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.093992    5600 scope.go:122] "RemoveContainer" containerID="2c08a215b4c47d1c45332b43ef6b7370279304482487375eb6949ab3817c4802"
Nov 03 20:15:44.094376 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:44.094355    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk_odh-ai-gateway-infra(3c35eed5-b7e1-4792-838e-cfde3d6de8d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9"
Nov 03 20:15:44.096218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.096193    5600 generic.go:356] "Generic (PLEG): container finished" podID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerID="fef4ec10757c915dae968c54b3e4fe70e27fcacb301d328837b73f97f0def831" exitCode=60
Nov 03 20:15:44.096322 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.096278    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" event={"ID":"c8c6716a-c78a-4aa8-9ead-eba203f65cc1","Type":"ContainerDied","Data":"fef4ec10757c915dae968c54b3e4fe70e27fcacb301d328837b73f97f0def831"}
Nov 03 20:15:44.096375 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.096325    5600 scope.go:122] "RemoveContainer" containerID="2b92515a6b9edf3509385061e1114fac3bda2c4f2fecc016ec7b390ec7da433d"
Nov 03 20:15:44.096616 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.096594    5600 scope.go:122] "RemoveContainer" containerID="fef4ec10757c915dae968c54b3e4fe70e27fcacb301d328837b73f97f0def831"
Nov 03 20:15:44.097125 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:44.096974    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97_odh-ai-gateway-infra(c8c6716a-c78a-4aa8-9ead-eba203f65cc1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1"
Nov 03 20:15:44.296098 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.296066    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:44.328828 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.328790    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"]
Nov 03 20:15:44.329943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.329916    5600 cpu_manager.go:414] "containerMap: removing container" podUID="36123290-c071-43cd-8731-a44a4d12597b" containerName="cleanup"
Nov 03 20:15:44.329943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.329942    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36123290-c071-43cd-8731-a44a4d12597b" containerName="cleanup"
Nov 03 20:15:44.330148 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.329994    5600 cpu_manager.go:414] "containerMap: removing container" podUID="fd38fad9-7ea8-4f12-8b53-747ed99e901b" containerName="revoke-keys"
Nov 03 20:15:44.330148 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.330001    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd38fad9-7ea8-4f12-8b53-747ed99e901b" containerName="revoke-keys"
Nov 03 20:15:44.330279 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.330185    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd38fad9-7ea8-4f12-8b53-747ed99e901b" containerName="revoke-keys"
Nov 03 20:15:44.330279 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.330211    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="36123290-c071-43cd-8731-a44a4d12597b" containerName="cleanup"
Nov 03 20:15:44.335956 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.335928    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"
Nov 03 20:15:44.340878 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.340754    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"]
Nov 03 20:15:44.372178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.372140    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-xks-ca\") pod \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") "
Nov 03 20:15:44.372335 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.372231    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd38fad9-7ea8-4f12-8b53-747ed99e901b-kube-api-access-84m2d\" (UniqueName: \"kubernetes.io/projected/fd38fad9-7ea8-4f12-8b53-747ed99e901b-kube-api-access-84m2d\") pod \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") "
Nov 03 20:15:44.372421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.372345    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-openshift-ca\") pod \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\" (UID: \"fd38fad9-7ea8-4f12-8b53-747ed99e901b\") "
Nov 03 20:15:44.372804 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.372772    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-openshift-ca" pod "fd38fad9-7ea8-4f12-8b53-747ed99e901b" (UID: "fd38fad9-7ea8-4f12-8b53-747ed99e901b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:44.373287 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.373258    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbm8\" (UniqueName: \"kubernetes.io/projected/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-kube-api-access-xvbm8\") pod \"maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k\" (UID: \"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"
Nov 03 20:15:44.373414 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.373378    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k\" (UID: \"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"
Nov 03 20:15:44.373653 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.373613    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:44.376884 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.376855    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-xks-ca" pod "fd38fad9-7ea8-4f12-8b53-747ed99e901b" (UID: "fd38fad9-7ea8-4f12-8b53-747ed99e901b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:15:44.377009 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.376984    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd38fad9-7ea8-4f12-8b53-747ed99e901b-kube-api-access-84m2d" pod "fd38fad9-7ea8-4f12-8b53-747ed99e901b" (UID: "fd38fad9-7ea8-4f12-8b53-747ed99e901b"). InnerVolumeSpecName "kube-api-access-84m2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:44.499229 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.481582    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbm8\" (UniqueName: \"kubernetes.io/projected/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-kube-api-access-xvbm8\") pod \"maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k\" (UID: \"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"
Nov 03 20:15:44.499229 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.481760    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k\" (UID: \"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"
Nov 03 20:15:44.499229 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.482093    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd38fad9-7ea8-4f12-8b53-747ed99e901b-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:44.499229 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.482119    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84m2d\" (UniqueName: \"kubernetes.io/projected/fd38fad9-7ea8-4f12-8b53-747ed99e901b-kube-api-access-84m2d\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:44.499229 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.483107    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k\" (UID: \"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"
Nov 03 20:15:44.505416 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.505380    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbm8\" (UniqueName: \"kubernetes.io/projected/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-kube-api-access-xvbm8\") pod \"maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k\" (UID: \"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"
Nov 03 20:15:44.660872 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.660839    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"
Nov 03 20:15:44.955954 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:44.955925    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"]
Nov 03 20:15:44.956084 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:44.956029    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb78ddb_9de0_4f4b_afe7_f5ae8721ebcc.slice/crio-a62d51774f597211772baaba12e71b40fb5c0650963facfaffd291579e24e28f WatchSource:0}: Error finding container a62d51774f597211772baaba12e71b40fb5c0650963facfaffd291579e24e28f: Status 404 returned error can't find the container with id a62d51774f597211772baaba12e71b40fb5c0650963facfaffd291579e24e28f
Nov 03 20:15:45.147565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:45.147420    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"
Nov 03 20:15:45.148038 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:45.147491    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc" event={"ID":"fd38fad9-7ea8-4f12-8b53-747ed99e901b","Type":"ContainerDied","Data":"62366fee57f630c2553c5254c855f54abba7daa4bff9987c7081dfd43c629314"}
Nov 03 20:15:45.148088 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:45.148048    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62366fee57f630c2553c5254c855f54abba7daa4bff9987c7081dfd43c629314"
Nov 03 20:15:45.150549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:45.150528    5600 scope.go:122] "RemoveContainer" containerID="fef4ec10757c915dae968c54b3e4fe70e27fcacb301d328837b73f97f0def831"
Nov 03 20:15:45.150938 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:45.150908    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97_odh-ai-gateway-infra(c8c6716a-c78a-4aa8-9ead-eba203f65cc1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1"
Nov 03 20:15:45.151857 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:45.151832    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k" event={"ID":"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc","Type":"ContainerStarted","Data":"a62d51774f597211772baaba12e71b40fb5c0650963facfaffd291579e24e28f"}
Nov 03 20:15:46.167460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:46.166234    5600 generic.go:356] "Generic (PLEG): container finished" podID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerID="8f902ba4c1a487c8c7e8e32f1ad840532dbe35e5485fa72fc8abc1b727a021f3" exitCode=60
Nov 03 20:15:46.167460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:46.166533    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k" event={"ID":"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc","Type":"ContainerDied","Data":"8f902ba4c1a487c8c7e8e32f1ad840532dbe35e5485fa72fc8abc1b727a021f3"}
Nov 03 20:15:46.167460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:46.167166    5600 scope.go:122] "RemoveContainer" containerID="8f902ba4c1a487c8c7e8e32f1ad840532dbe35e5485fa72fc8abc1b727a021f3"
Nov 03 20:15:47.205056 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:47.205021    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k" event={"ID":"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc","Type":"ContainerStarted","Data":"410238442d8d3fa04c2132a0f3382e682932cdc53128be4bca1c39f575b2862e"}
Nov 03 20:15:47.269664 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:47.269600    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k" podStartSLOduration=47.269568008 podStartE2EDuration="47.269568008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:47.248907049 +0000 UTC m=+2129.761728832" watchObservedRunningTime="2026-11-03 20:15:47.269568008 +0000 UTC m=+2129.782389792"
Nov 03 20:15:48.217776 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:48.217737    5600 generic.go:356] "Generic (PLEG): container finished" podID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerID="410238442d8d3fa04c2132a0f3382e682932cdc53128be4bca1c39f575b2862e" exitCode=60
Nov 03 20:15:48.218292 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:48.217825    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k" event={"ID":"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc","Type":"ContainerDied","Data":"410238442d8d3fa04c2132a0f3382e682932cdc53128be4bca1c39f575b2862e"}
Nov 03 20:15:48.218292 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:48.217896    5600 scope.go:122] "RemoveContainer" containerID="8f902ba4c1a487c8c7e8e32f1ad840532dbe35e5485fa72fc8abc1b727a021f3"
Nov 03 20:15:48.218292 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:48.218223    5600 scope.go:122] "RemoveContainer" containerID="410238442d8d3fa04c2132a0f3382e682932cdc53128be4bca1c39f575b2862e"
Nov 03 20:15:48.218712 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:48.218689    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k_odh-ai-gateway-infra(abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc"
Nov 03 20:15:49.228832 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:49.228750    5600 scope.go:122] "RemoveContainer" containerID="410238442d8d3fa04c2132a0f3382e682932cdc53128be4bca1c39f575b2862e"
Nov 03 20:15:49.229262 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:49.229219    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k_odh-ai-gateway-infra(abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc"
Nov 03 20:15:52.609911 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:52.609875    5600 scope.go:122] "RemoveContainer" containerID="2c08a215b4c47d1c45332b43ef6b7370279304482487375eb6949ab3817c4802"
Nov 03 20:15:53.269658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:53.269561    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" event={"ID":"3c35eed5-b7e1-4792-838e-cfde3d6de8d9","Type":"ContainerStarted","Data":"b18925d841f669bc6a8329b7f1de15a6cc5e4a3440d5695f85de35a43d5d6139"}
Nov 03 20:15:53.613069 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:53.612980    5600 scope.go:122] "RemoveContainer" containerID="6e6bf238301f4c8efc3abae3ce79cbb3a421df8e9f2f44a6b6a8f1c7e39a8d8d"
Nov 03 20:15:54.284704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:54.284611    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" event={"ID":"ba818408-2dbb-489d-b101-ca955de8aedf","Type":"ContainerStarted","Data":"8f1f9b2f346cbe0b34b7fef849c21abc8a592a2342fa11e329aba749210f41b5"}
Nov 03 20:15:54.288310 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:54.288276    5600 generic.go:356] "Generic (PLEG): container finished" podID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerID="b18925d841f669bc6a8329b7f1de15a6cc5e4a3440d5695f85de35a43d5d6139" exitCode=60
Nov 03 20:15:54.288542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:54.288356    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" event={"ID":"3c35eed5-b7e1-4792-838e-cfde3d6de8d9","Type":"ContainerDied","Data":"b18925d841f669bc6a8329b7f1de15a6cc5e4a3440d5695f85de35a43d5d6139"}
Nov 03 20:15:54.288542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:54.288418    5600 scope.go:122] "RemoveContainer" containerID="2c08a215b4c47d1c45332b43ef6b7370279304482487375eb6949ab3817c4802"
Nov 03 20:15:54.288991 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:54.288970    5600 scope.go:122] "RemoveContainer" containerID="b18925d841f669bc6a8329b7f1de15a6cc5e4a3440d5695f85de35a43d5d6139"
Nov 03 20:15:54.289417 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:54.289390    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk_odh-ai-gateway-infra(3c35eed5-b7e1-4792-838e-cfde3d6de8d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9"
Nov 03 20:15:54.304037 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:54.302942    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"]
Nov 03 20:15:54.611082 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:54.610500    5600 scope.go:122] "RemoveContainer" containerID="fef4ec10757c915dae968c54b3e4fe70e27fcacb301d328837b73f97f0def831"
Nov 03 20:15:55.298887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.298856    5600 generic.go:356] "Generic (PLEG): container finished" podID="ba818408-2dbb-489d-b101-ca955de8aedf" containerID="8f1f9b2f346cbe0b34b7fef849c21abc8a592a2342fa11e329aba749210f41b5" exitCode=60
Nov 03 20:15:55.299354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.298931    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" event={"ID":"ba818408-2dbb-489d-b101-ca955de8aedf","Type":"ContainerDied","Data":"8f1f9b2f346cbe0b34b7fef849c21abc8a592a2342fa11e329aba749210f41b5"}
Nov 03 20:15:55.299354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.298993    5600 scope.go:122] "RemoveContainer" containerID="6e6bf238301f4c8efc3abae3ce79cbb3a421df8e9f2f44a6b6a8f1c7e39a8d8d"
Nov 03 20:15:55.299518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.299483    5600 scope.go:122] "RemoveContainer" containerID="8f1f9b2f346cbe0b34b7fef849c21abc8a592a2342fa11e329aba749210f41b5"
Nov 03 20:15:55.299895 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:55.299872    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp_odh-ai-gateway-infra(ba818408-2dbb-489d-b101-ca955de8aedf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" podUID="ba818408-2dbb-489d-b101-ca955de8aedf"
Nov 03 20:15:55.303172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.303145    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" event={"ID":"c8c6716a-c78a-4aa8-9ead-eba203f65cc1","Type":"ContainerStarted","Data":"68e5593f5690031417e3b3a30e7dbf21c627488f4e5d1b8af0eee970c36f2c56"}
Nov 03 20:15:55.326531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.326496    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"]
Nov 03 20:15:55.502317 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.502281    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"
Nov 03 20:15:55.537969 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.537934    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"]
Nov 03 20:15:55.538745 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.538711    5600 cpu_manager.go:414] "containerMap: removing container" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerName="cleanup"
Nov 03 20:15:55.538745 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.538746    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerName="cleanup"
Nov 03 20:15:55.538927 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.538758    5600 cpu_manager.go:414] "containerMap: removing container" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerName="cleanup"
Nov 03 20:15:55.538927 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.538762    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerName="cleanup"
Nov 03 20:15:55.538927 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.538926    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerName="cleanup"
Nov 03 20:15:55.539092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.538935    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerName="cleanup"
Nov 03 20:15:55.539092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.538943    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerName="cleanup"
Nov 03 20:15:55.540500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.540468    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-maas-api-service-ca\") pod \"3c35eed5-b7e1-4792-838e-cfde3d6de8d9\" (UID: \"3c35eed5-b7e1-4792-838e-cfde3d6de8d9\") "
Nov 03 20:15:55.540747 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.540729    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-kube-api-access-wwshm\" (UniqueName: \"kubernetes.io/projected/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-kube-api-access-wwshm\") pod \"3c35eed5-b7e1-4792-838e-cfde3d6de8d9\" (UID: \"3c35eed5-b7e1-4792-838e-cfde3d6de8d9\") "
Nov 03 20:15:55.540950 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.540925    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-maas-api-service-ca" pod "3c35eed5-b7e1-4792-838e-cfde3d6de8d9" (UID: "3c35eed5-b7e1-4792-838e-cfde3d6de8d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:55.541483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.541401    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:55.554216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.554192    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"
Nov 03 20:15:55.555725 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.554069    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"]
Nov 03 20:15:55.556314 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.556290    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-kube-api-access-wwshm" pod "3c35eed5-b7e1-4792-838e-cfde3d6de8d9" (UID: "3c35eed5-b7e1-4792-838e-cfde3d6de8d9"). InnerVolumeSpecName "kube-api-access-wwshm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:55.642582 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.642542    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptmxt\" (UniqueName: \"kubernetes.io/projected/2bf21193-3ee2-451e-b37b-6b4a99a08c35-kube-api-access-ptmxt\") pod \"maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n\" (UID: \"2bf21193-3ee2-451e-b37b-6b4a99a08c35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"
Nov 03 20:15:55.642704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.642682    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf21193-3ee2-451e-b37b-6b4a99a08c35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n\" (UID: \"2bf21193-3ee2-451e-b37b-6b4a99a08c35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"
Nov 03 20:15:55.643532 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.643404    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwshm\" (UniqueName: \"kubernetes.io/projected/3c35eed5-b7e1-4792-838e-cfde3d6de8d9-kube-api-access-wwshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:55.744496 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.744398    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptmxt\" (UniqueName: \"kubernetes.io/projected/2bf21193-3ee2-451e-b37b-6b4a99a08c35-kube-api-access-ptmxt\") pod \"maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n\" (UID: \"2bf21193-3ee2-451e-b37b-6b4a99a08c35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"
Nov 03 20:15:55.744674 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.744521    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf21193-3ee2-451e-b37b-6b4a99a08c35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n\" (UID: \"2bf21193-3ee2-451e-b37b-6b4a99a08c35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"
Nov 03 20:15:55.745176 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.745149    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf21193-3ee2-451e-b37b-6b4a99a08c35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n\" (UID: \"2bf21193-3ee2-451e-b37b-6b4a99a08c35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"
Nov 03 20:15:55.753233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.753203    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptmxt\" (UniqueName: \"kubernetes.io/projected/2bf21193-3ee2-451e-b37b-6b4a99a08c35-kube-api-access-ptmxt\") pod \"maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n\" (UID: \"2bf21193-3ee2-451e-b37b-6b4a99a08c35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"
Nov 03 20:15:55.890729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:55.890691    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"
Nov 03 20:15:56.297772 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:56.297736    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf21193_3ee2_451e_b37b_6b4a99a08c35.slice/crio-3ba943ef1ccfef17a60dd03f364a82f2fe134f15e6605e327ffb1c44a616e770 WatchSource:0}: Error finding container 3ba943ef1ccfef17a60dd03f364a82f2fe134f15e6605e327ffb1c44a616e770: Status 404 returned error can't find the container with id 3ba943ef1ccfef17a60dd03f364a82f2fe134f15e6605e327ffb1c44a616e770
Nov 03 20:15:56.300945 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.300905    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"]
Nov 03 20:15:56.316986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.316917    5600 generic.go:356] "Generic (PLEG): container finished" podID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerID="68e5593f5690031417e3b3a30e7dbf21c627488f4e5d1b8af0eee970c36f2c56" exitCode=60
Nov 03 20:15:56.317102 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.316998    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" event={"ID":"c8c6716a-c78a-4aa8-9ead-eba203f65cc1","Type":"ContainerDied","Data":"68e5593f5690031417e3b3a30e7dbf21c627488f4e5d1b8af0eee970c36f2c56"}
Nov 03 20:15:56.317102 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.317078    5600 scope.go:122] "RemoveContainer" containerID="fef4ec10757c915dae968c54b3e4fe70e27fcacb301d328837b73f97f0def831"
Nov 03 20:15:56.317607 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.317588    5600 scope.go:122] "RemoveContainer" containerID="68e5593f5690031417e3b3a30e7dbf21c627488f4e5d1b8af0eee970c36f2c56"
Nov 03 20:15:56.318029 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:56.317996    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97_odh-ai-gateway-infra(c8c6716a-c78a-4aa8-9ead-eba203f65cc1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1"
Nov 03 20:15:56.320592 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.319946    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n" event={"ID":"2bf21193-3ee2-451e-b37b-6b4a99a08c35","Type":"ContainerStarted","Data":"3ba943ef1ccfef17a60dd03f364a82f2fe134f15e6605e327ffb1c44a616e770"}
Nov 03 20:15:56.323814 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.323790    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk" event={"ID":"3c35eed5-b7e1-4792-838e-cfde3d6de8d9","Type":"ContainerDied","Data":"b4d8434cc9688786492b4f4f6f6cde4e73ac56f46bfd374bb158731a144d140f"}
Nov 03 20:15:56.323920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.323812    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"
Nov 03 20:15:56.364903 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.364877    5600 scope.go:122] "RemoveContainer" containerID="b18925d841f669bc6a8329b7f1de15a6cc5e4a3440d5695f85de35a43d5d6139"
Nov 03 20:15:56.401514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.401454    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"]
Nov 03 20:15:56.406872 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.406840    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b1aed-29895615-qxvfk"]
Nov 03 20:15:56.467505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.467336    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"]
Nov 03 20:15:56.561335 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.561306    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"
Nov 03 20:15:56.569938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.569870    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba818408-2dbb-489d-b101-ca955de8aedf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba818408-2dbb-489d-b101-ca955de8aedf-maas-api-service-ca\") pod \"ba818408-2dbb-489d-b101-ca955de8aedf\" (UID: \"ba818408-2dbb-489d-b101-ca955de8aedf\") "
Nov 03 20:15:56.570173 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.570145    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba818408-2dbb-489d-b101-ca955de8aedf-kube-api-access-5w75g\" (UniqueName: \"kubernetes.io/projected/ba818408-2dbb-489d-b101-ca955de8aedf-kube-api-access-5w75g\") pod \"ba818408-2dbb-489d-b101-ca955de8aedf\" (UID: \"ba818408-2dbb-489d-b101-ca955de8aedf\") "
Nov 03 20:15:56.570319 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.570294    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba818408-2dbb-489d-b101-ca955de8aedf-maas-api-service-ca" pod "ba818408-2dbb-489d-b101-ca955de8aedf" (UID: "ba818408-2dbb-489d-b101-ca955de8aedf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:56.570897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.570875    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba818408-2dbb-489d-b101-ca955de8aedf-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:56.572519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.572497    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba818408-2dbb-489d-b101-ca955de8aedf-kube-api-access-5w75g" pod "ba818408-2dbb-489d-b101-ca955de8aedf" (UID: "ba818408-2dbb-489d-b101-ca955de8aedf"). InnerVolumeSpecName "kube-api-access-5w75g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:56.597412 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.597329    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"]
Nov 03 20:15:56.598567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.598494    5600 cpu_manager.go:414] "containerMap: removing container" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerName="cleanup"
Nov 03 20:15:56.598567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.598514    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" containerName="cleanup"
Nov 03 20:15:56.598567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.598566    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ba818408-2dbb-489d-b101-ca955de8aedf" containerName="cleanup"
Nov 03 20:15:56.598773 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.598575    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba818408-2dbb-489d-b101-ca955de8aedf" containerName="cleanup"
Nov 03 20:15:56.598773 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.598615    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ba818408-2dbb-489d-b101-ca955de8aedf" containerName="cleanup"
Nov 03 20:15:56.598773 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.598622    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba818408-2dbb-489d-b101-ca955de8aedf" containerName="cleanup"
Nov 03 20:15:56.598923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.598808    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba818408-2dbb-489d-b101-ca955de8aedf" containerName="cleanup"
Nov 03 20:15:56.598923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.598821    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba818408-2dbb-489d-b101-ca955de8aedf" containerName="cleanup"
Nov 03 20:15:56.598923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.598843    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba818408-2dbb-489d-b101-ca955de8aedf" containerName="cleanup"
Nov 03 20:15:56.604053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.603652    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"
Nov 03 20:15:56.613413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.613382    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"]
Nov 03 20:15:56.672404 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.672360    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npxw\" (UniqueName: \"kubernetes.io/projected/256aebcd-7912-4b33-817e-53b38c027027-kube-api-access-4npxw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp\" (UID: \"256aebcd-7912-4b33-817e-53b38c027027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"
Nov 03 20:15:56.672746 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.672720    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/256aebcd-7912-4b33-817e-53b38c027027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp\" (UID: \"256aebcd-7912-4b33-817e-53b38c027027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"
Nov 03 20:15:56.672926 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.672899    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w75g\" (UniqueName: \"kubernetes.io/projected/ba818408-2dbb-489d-b101-ca955de8aedf-kube-api-access-5w75g\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:56.773819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.773768    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/256aebcd-7912-4b33-817e-53b38c027027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp\" (UID: \"256aebcd-7912-4b33-817e-53b38c027027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"
Nov 03 20:15:56.774000 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.773939    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npxw\" (UniqueName: \"kubernetes.io/projected/256aebcd-7912-4b33-817e-53b38c027027-kube-api-access-4npxw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp\" (UID: \"256aebcd-7912-4b33-817e-53b38c027027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"
Nov 03 20:15:56.774535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.774513    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/256aebcd-7912-4b33-817e-53b38c027027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp\" (UID: \"256aebcd-7912-4b33-817e-53b38c027027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"
Nov 03 20:15:56.783226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.783194    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npxw\" (UniqueName: \"kubernetes.io/projected/256aebcd-7912-4b33-817e-53b38c027027-kube-api-access-4npxw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp\" (UID: \"256aebcd-7912-4b33-817e-53b38c027027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"
Nov 03 20:15:56.932311 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:56.932280    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"
Nov 03 20:15:57.220792 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:57.220750    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256aebcd_7912_4b33_817e_53b38c027027.slice/crio-165485b40c1cfc6525eae1b8a75c13c8f4bf3f1ac4c110e64b8e63b026308c58 WatchSource:0}: Error finding container 165485b40c1cfc6525eae1b8a75c13c8f4bf3f1ac4c110e64b8e63b026308c58: Status 404 returned error can't find the container with id 165485b40c1cfc6525eae1b8a75c13c8f4bf3f1ac4c110e64b8e63b026308c58
Nov 03 20:15:57.223649 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.223496    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"]
Nov 03 20:15:57.335267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.335226    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n" event={"ID":"2bf21193-3ee2-451e-b37b-6b4a99a08c35","Type":"ContainerStarted","Data":"8b81c84b2669564fb83f903128abcb8d3a632fa673912bb46fc640ac36944ced"}
Nov 03 20:15:57.337003 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.336972    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp" event={"ID":"ba818408-2dbb-489d-b101-ca955de8aedf","Type":"ContainerDied","Data":"91deafac96d8a7e3b775802ef37ac4c3696759df232a8427727b36ccfb40d9c5"}
Nov 03 20:15:57.337167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.336980    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"
Nov 03 20:15:57.337239 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.337029    5600 scope.go:122] "RemoveContainer" containerID="8f1f9b2f346cbe0b34b7fef849c21abc8a592a2342fa11e329aba749210f41b5"
Nov 03 20:15:57.341328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.341302    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp" event={"ID":"256aebcd-7912-4b33-817e-53b38c027027","Type":"ContainerStarted","Data":"165485b40c1cfc6525eae1b8a75c13c8f4bf3f1ac4c110e64b8e63b026308c58"}
Nov 03 20:15:57.396461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.384192    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n" podStartSLOduration=57.384168378 podStartE2EDuration="57.384168378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:57.352640259 +0000 UTC m=+2139.865462041" watchObservedRunningTime="2026-11-03 20:15:57.384168378 +0000 UTC m=+2139.896990158"
Nov 03 20:15:57.426176 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.426134    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"]
Nov 03 20:15:57.435501 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.434852    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d81d18-29895615-llngp"]
Nov 03 20:15:57.596687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.596656    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"
Nov 03 20:15:57.633544 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.633504    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c35eed5-b7e1-4792-838e-cfde3d6de8d9" path="/var/lib/kubelet/pods/3c35eed5-b7e1-4792-838e-cfde3d6de8d9/volumes"
Nov 03 20:15:57.634082 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.634062    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba818408-2dbb-489d-b101-ca955de8aedf" path="/var/lib/kubelet/pods/ba818408-2dbb-489d-b101-ca955de8aedf/volumes"
Nov 03 20:15:57.636919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.636889    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"]
Nov 03 20:15:57.637721 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.637700    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ba818408-2dbb-489d-b101-ca955de8aedf" containerName="cleanup"
Nov 03 20:15:57.637885 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.637867    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba818408-2dbb-489d-b101-ca955de8aedf" containerName="cleanup"
Nov 03 20:15:57.638298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.638255    5600 cpu_manager.go:414] "containerMap: removing container" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerName="cleanup"
Nov 03 20:15:57.638298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.638279    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerName="cleanup"
Nov 03 20:15:57.638298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.638296    5600 cpu_manager.go:414] "containerMap: removing container" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerName="cleanup"
Nov 03 20:15:57.638298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.638303    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerName="cleanup"
Nov 03 20:15:57.638623 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.638340    5600 cpu_manager.go:414] "containerMap: removing container" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerName="cleanup"
Nov 03 20:15:57.638623 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.638347    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerName="cleanup"
Nov 03 20:15:57.638623 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.638605    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerName="cleanup"
Nov 03 20:15:57.638623 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.638621    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerName="cleanup"
Nov 03 20:15:57.638896 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.638635    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" containerName="cleanup"
Nov 03 20:15:57.644290 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.644264    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"
Nov 03 20:15:57.652696 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.652670    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"]
Nov 03 20:15:57.710686 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.710647    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-kube-api-access-rzk5n\" (UniqueName: \"kubernetes.io/projected/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-kube-api-access-rzk5n\") pod \"c8c6716a-c78a-4aa8-9ead-eba203f65cc1\" (UID: \"c8c6716a-c78a-4aa8-9ead-eba203f65cc1\") "
Nov 03 20:15:57.711020 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.710978    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-maas-api-service-ca\") pod \"c8c6716a-c78a-4aa8-9ead-eba203f65cc1\" (UID: \"c8c6716a-c78a-4aa8-9ead-eba203f65cc1\") "
Nov 03 20:15:57.711396 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.711355    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-maas-api-service-ca" pod "c8c6716a-c78a-4aa8-9ead-eba203f65cc1" (UID: "c8c6716a-c78a-4aa8-9ead-eba203f65cc1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:57.711724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.711477    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tsf7\" (UniqueName: \"kubernetes.io/projected/b2cdaecf-78a7-4665-9900-20d91cd8c806-kube-api-access-2tsf7\") pod \"maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn\" (UID: \"b2cdaecf-78a7-4665-9900-20d91cd8c806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"
Nov 03 20:15:57.711724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.711580    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cdaecf-78a7-4665-9900-20d91cd8c806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn\" (UID: \"b2cdaecf-78a7-4665-9900-20d91cd8c806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"
Nov 03 20:15:57.712003 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.711980    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:57.713324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.713284    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-kube-api-access-rzk5n" pod "c8c6716a-c78a-4aa8-9ead-eba203f65cc1" (UID: "c8c6716a-c78a-4aa8-9ead-eba203f65cc1"). InnerVolumeSpecName "kube-api-access-rzk5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:57.813524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.813400    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tsf7\" (UniqueName: \"kubernetes.io/projected/b2cdaecf-78a7-4665-9900-20d91cd8c806-kube-api-access-2tsf7\") pod \"maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn\" (UID: \"b2cdaecf-78a7-4665-9900-20d91cd8c806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"
Nov 03 20:15:57.813524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.813519    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cdaecf-78a7-4665-9900-20d91cd8c806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn\" (UID: \"b2cdaecf-78a7-4665-9900-20d91cd8c806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"
Nov 03 20:15:57.813785 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.813734    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzk5n\" (UniqueName: \"kubernetes.io/projected/c8c6716a-c78a-4aa8-9ead-eba203f65cc1-kube-api-access-rzk5n\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:57.814200 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.814174    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cdaecf-78a7-4665-9900-20d91cd8c806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn\" (UID: \"b2cdaecf-78a7-4665-9900-20d91cd8c806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"
Nov 03 20:15:57.821363 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.821336    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tsf7\" (UniqueName: \"kubernetes.io/projected/b2cdaecf-78a7-4665-9900-20d91cd8c806-kube-api-access-2tsf7\") pod \"maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn\" (UID: \"b2cdaecf-78a7-4665-9900-20d91cd8c806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"
Nov 03 20:15:57.966561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:57.966522    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"
Nov 03 20:15:58.301835 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:15:58.301796    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cdaecf_78a7_4665_9900_20d91cd8c806.slice/crio-754de1847a9867d348ebb44ee0f771bd041477e33abacbbb8aac400e4a3b9f2e WatchSource:0}: Error finding container 754de1847a9867d348ebb44ee0f771bd041477e33abacbbb8aac400e4a3b9f2e: Status 404 returned error can't find the container with id 754de1847a9867d348ebb44ee0f771bd041477e33abacbbb8aac400e4a3b9f2e
Nov 03 20:15:58.302078 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.302047    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"]
Nov 03 20:15:58.353896 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.353862    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"
Nov 03 20:15:58.353896 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.353882    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97" event={"ID":"c8c6716a-c78a-4aa8-9ead-eba203f65cc1","Type":"ContainerDied","Data":"749a0fdbc50f363f07965fbd32c0878145da9409e617c30f38b347a3c84ece0f"}
Nov 03 20:15:58.354379 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.353959    5600 scope.go:122] "RemoveContainer" containerID="68e5593f5690031417e3b3a30e7dbf21c627488f4e5d1b8af0eee970c36f2c56"
Nov 03 20:15:58.356419 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.356389    5600 generic.go:356] "Generic (PLEG): container finished" podID="2bf21193-3ee2-451e-b37b-6b4a99a08c35" containerID="8b81c84b2669564fb83f903128abcb8d3a632fa673912bb46fc640ac36944ced" exitCode=60
Nov 03 20:15:58.356597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.356463    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n" event={"ID":"2bf21193-3ee2-451e-b37b-6b4a99a08c35","Type":"ContainerDied","Data":"8b81c84b2669564fb83f903128abcb8d3a632fa673912bb46fc640ac36944ced"}
Nov 03 20:15:58.356831 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.356813    5600 scope.go:122] "RemoveContainer" containerID="8b81c84b2669564fb83f903128abcb8d3a632fa673912bb46fc640ac36944ced"
Nov 03 20:15:58.359647 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.359620    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn" event={"ID":"b2cdaecf-78a7-4665-9900-20d91cd8c806","Type":"ContainerStarted","Data":"754de1847a9867d348ebb44ee0f771bd041477e33abacbbb8aac400e4a3b9f2e"}
Nov 03 20:15:58.361912 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.361889    5600 generic.go:356] "Generic (PLEG): container finished" podID="256aebcd-7912-4b33-817e-53b38c027027" containerID="bf30e1815e08cb7e89b9d1346ee52862cac5994d3b4699823e7248281f7d5f49" exitCode=60
Nov 03 20:15:58.362017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.361989    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp" event={"ID":"256aebcd-7912-4b33-817e-53b38c027027","Type":"ContainerDied","Data":"bf30e1815e08cb7e89b9d1346ee52862cac5994d3b4699823e7248281f7d5f49"}
Nov 03 20:15:58.362267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.362251    5600 scope.go:122] "RemoveContainer" containerID="bf30e1815e08cb7e89b9d1346ee52862cac5994d3b4699823e7248281f7d5f49"
Nov 03 20:15:58.418095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.418062    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"]
Nov 03 20:15:58.422758 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.422728    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cad531-29895615-k2n97"]
Nov 03 20:15:58.609952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:58.609878    5600 scope.go:122] "RemoveContainer" containerID="410238442d8d3fa04c2132a0f3382e682932cdc53128be4bca1c39f575b2862e"
Nov 03 20:15:58.815859 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:58.815822    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:15:59.374202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:59.374156    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k" event={"ID":"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc","Type":"ContainerStarted","Data":"fa397ba87210423558e67de8535b3bae0db397c3930b3e94eef5e82a3fcf47ee"}
Nov 03 20:15:59.376256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:59.376220    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n" event={"ID":"2bf21193-3ee2-451e-b37b-6b4a99a08c35","Type":"ContainerStarted","Data":"afeaa2bd72257435c8e9cade737f69f9aa6c275e4d60e548262b693e9dc9d5ba"}
Nov 03 20:15:59.377914 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:59.377892    5600 generic.go:356] "Generic (PLEG): container finished" podID="b2cdaecf-78a7-4665-9900-20d91cd8c806" containerID="24895f93c7caaf0e1f5fa8430d6dc86f91bbe9130f19e81a572f5e9cf2203e56" exitCode=60
Nov 03 20:15:59.378017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:59.377957    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn" event={"ID":"b2cdaecf-78a7-4665-9900-20d91cd8c806","Type":"ContainerDied","Data":"24895f93c7caaf0e1f5fa8430d6dc86f91bbe9130f19e81a572f5e9cf2203e56"}
Nov 03 20:15:59.387246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:59.378914    5600 scope.go:122] "RemoveContainer" containerID="24895f93c7caaf0e1f5fa8430d6dc86f91bbe9130f19e81a572f5e9cf2203e56"
Nov 03 20:15:59.397704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:59.397660    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp" event={"ID":"256aebcd-7912-4b33-817e-53b38c027027","Type":"ContainerStarted","Data":"ee3de33aa3bfd7487287b80ad668e5c683829bc4f9b8e324745f16c25294f496"}
Nov 03 20:15:59.457680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:59.457572    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp" podStartSLOduration=59.457553348 podStartE2EDuration="59.457553348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:59.446405699 +0000 UTC m=+2141.959227481" watchObservedRunningTime="2026-11-03 20:15:59.457553348 +0000 UTC m=+2141.970375129"
Nov 03 20:15:59.622073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:15:59.622038    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c6716a-c78a-4aa8-9ead-eba203f65cc1" path="/var/lib/kubelet/pods/c8c6716a-c78a-4aa8-9ead-eba203f65cc1/volumes"
Nov 03 20:15:59.827177 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:15:59.827143    5600 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256aebcd_7912_4b33_817e_53b38c027027.slice/crio-ee3de33aa3bfd7487287b80ad668e5c683829bc4f9b8e324745f16c25294f496.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:16:00.023800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.023766    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"]
Nov 03 20:16:00.053192 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.053155    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"]
Nov 03 20:16:00.059936 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.059892    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"]
Nov 03 20:16:00.137473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.137366    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"]
Nov 03 20:16:00.410483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.410342    5600 generic.go:356] "Generic (PLEG): container finished" podID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerID="fa397ba87210423558e67de8535b3bae0db397c3930b3e94eef5e82a3fcf47ee" exitCode=60
Nov 03 20:16:00.410963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.410490    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k" event={"ID":"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc","Type":"ContainerDied","Data":"fa397ba87210423558e67de8535b3bae0db397c3930b3e94eef5e82a3fcf47ee"}
Nov 03 20:16:00.410963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.410568    5600 scope.go:122] "RemoveContainer" containerID="410238442d8d3fa04c2132a0f3382e682932cdc53128be4bca1c39f575b2862e"
Nov 03 20:16:00.412803 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.412775    5600 generic.go:356] "Generic (PLEG): container finished" podID="2bf21193-3ee2-451e-b37b-6b4a99a08c35" containerID="afeaa2bd72257435c8e9cade737f69f9aa6c275e4d60e548262b693e9dc9d5ba" exitCode=60
Nov 03 20:16:00.412929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.412830    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n" event={"ID":"2bf21193-3ee2-451e-b37b-6b4a99a08c35","Type":"ContainerDied","Data":"afeaa2bd72257435c8e9cade737f69f9aa6c275e4d60e548262b693e9dc9d5ba"}
Nov 03 20:16:00.413097 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.413078    5600 scope.go:122] "RemoveContainer" containerID="afeaa2bd72257435c8e9cade737f69f9aa6c275e4d60e548262b693e9dc9d5ba"
Nov 03 20:16:00.413556 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:16:00.413522    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n_odh-ai-gateway-infra(2bf21193-3ee2-451e-b37b-6b4a99a08c35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n" podUID="2bf21193-3ee2-451e-b37b-6b4a99a08c35"
Nov 03 20:16:00.415338 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.415305    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn" event={"ID":"b2cdaecf-78a7-4665-9900-20d91cd8c806","Type":"ContainerStarted","Data":"da66d72560a5da9ad20918f5a282bea9e53d3690697125c6b2758f138603af07"}
Nov 03 20:16:00.418726 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.418696    5600 generic.go:356] "Generic (PLEG): container finished" podID="256aebcd-7912-4b33-817e-53b38c027027" containerID="ee3de33aa3bfd7487287b80ad668e5c683829bc4f9b8e324745f16c25294f496" exitCode=60
Nov 03 20:16:00.418905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.418882    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp" event={"ID":"256aebcd-7912-4b33-817e-53b38c027027","Type":"ContainerDied","Data":"ee3de33aa3bfd7487287b80ad668e5c683829bc4f9b8e324745f16c25294f496"}
Nov 03 20:16:00.419212 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.419194    5600 scope.go:122] "RemoveContainer" containerID="ee3de33aa3bfd7487287b80ad668e5c683829bc4f9b8e324745f16c25294f496"
Nov 03 20:16:00.419703 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:16:00.419675    5600 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp_odh-ai-gateway-infra(256aebcd-7912-4b33-817e-53b38c027027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp" podUID="256aebcd-7912-4b33-817e-53b38c027027"
Nov 03 20:16:00.427851 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.427827    5600 scope.go:122] "RemoveContainer" containerID="8b81c84b2669564fb83f903128abcb8d3a632fa673912bb46fc640ac36944ced"
Nov 03 20:16:00.446083 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.444944    5600 scope.go:122] "RemoveContainer" containerID="bf30e1815e08cb7e89b9d1346ee52862cac5994d3b4699823e7248281f7d5f49"
Nov 03 20:16:00.477963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.477912    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn" podStartSLOduration=60.477899759 podStartE2EDuration="1m0.477899759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:00.475156974 +0000 UTC m=+2142.987978757" watchObservedRunningTime="2026-11-03 20:16:00.477899759 +0000 UTC m=+2142.990721581"
Nov 03 20:16:00.646404 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.646375    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"
Nov 03 20:16:00.789848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.789748    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-maas-api-service-ca\") pod \"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc\" (UID: \"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc\") "
Nov 03 20:16:00.790056 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.790032    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-kube-api-access-xvbm8\" (UniqueName: \"kubernetes.io/projected/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-kube-api-access-xvbm8\") pod \"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc\" (UID: \"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc\") "
Nov 03 20:16:00.790151 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.790074    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-maas-api-service-ca" pod "abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" (UID: "abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:00.790672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.790649    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:00.792667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.792642    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-kube-api-access-xvbm8" pod "abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" (UID: "abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc"). InnerVolumeSpecName "kube-api-access-xvbm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:00.892709 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:00.892658    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbm8\" (UniqueName: \"kubernetes.io/projected/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc-kube-api-access-xvbm8\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:01.432016 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.431980    5600 generic.go:356] "Generic (PLEG): container finished" podID="b2cdaecf-78a7-4665-9900-20d91cd8c806" containerID="da66d72560a5da9ad20918f5a282bea9e53d3690697125c6b2758f138603af07" exitCode=60
Nov 03 20:16:01.432547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.432069    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn" event={"ID":"b2cdaecf-78a7-4665-9900-20d91cd8c806","Type":"ContainerDied","Data":"da66d72560a5da9ad20918f5a282bea9e53d3690697125c6b2758f138603af07"}
Nov 03 20:16:01.432547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.432190    5600 scope.go:122] "RemoveContainer" containerID="24895f93c7caaf0e1f5fa8430d6dc86f91bbe9130f19e81a572f5e9cf2203e56"
Nov 03 20:16:01.436621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.436600    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"
Nov 03 20:16:01.436729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.436635    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k" event={"ID":"abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc","Type":"ContainerDied","Data":"a62d51774f597211772baaba12e71b40fb5c0650963facfaffd291579e24e28f"}
Nov 03 20:16:01.453525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.452742    5600 scope.go:122] "RemoveContainer" containerID="fa397ba87210423558e67de8535b3bae0db397c3930b3e94eef5e82a3fcf47ee"
Nov 03 20:16:01.515336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.515303    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"]
Nov 03 20:16:01.561497 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.555489    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-facc79-29895615-hdb9k"]
Nov 03 20:16:01.662112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.662069    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" path="/var/lib/kubelet/pods/abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc/volumes"
Nov 03 20:16:01.855169 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.855142    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"
Nov 03 20:16:01.855982 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.855958    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"
Nov 03 20:16:01.864748 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:01.864723    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"
Nov 03 20:16:02.024624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.024592    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf21193-3ee2-451e-b37b-6b4a99a08c35-kube-api-access-ptmxt\" (UniqueName: \"kubernetes.io/projected/2bf21193-3ee2-451e-b37b-6b4a99a08c35-kube-api-access-ptmxt\") pod \"2bf21193-3ee2-451e-b37b-6b4a99a08c35\" (UID: \"2bf21193-3ee2-451e-b37b-6b4a99a08c35\") "
Nov 03 20:16:02.024830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.024640    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cdaecf-78a7-4665-9900-20d91cd8c806-kube-api-access-2tsf7\" (UniqueName: \"kubernetes.io/projected/b2cdaecf-78a7-4665-9900-20d91cd8c806-kube-api-access-2tsf7\") pod \"b2cdaecf-78a7-4665-9900-20d91cd8c806\" (UID: \"b2cdaecf-78a7-4665-9900-20d91cd8c806\") "
Nov 03 20:16:02.024830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.024718    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256aebcd-7912-4b33-817e-53b38c027027-kube-api-access-4npxw\" (UniqueName: \"kubernetes.io/projected/256aebcd-7912-4b33-817e-53b38c027027-kube-api-access-4npxw\") pod \"256aebcd-7912-4b33-817e-53b38c027027\" (UID: \"256aebcd-7912-4b33-817e-53b38c027027\") "
Nov 03 20:16:02.024830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.024788    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf21193-3ee2-451e-b37b-6b4a99a08c35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf21193-3ee2-451e-b37b-6b4a99a08c35-maas-api-service-ca\") pod \"2bf21193-3ee2-451e-b37b-6b4a99a08c35\" (UID: \"2bf21193-3ee2-451e-b37b-6b4a99a08c35\") "
Nov 03 20:16:02.025020 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.024931    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2cdaecf-78a7-4665-9900-20d91cd8c806-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cdaecf-78a7-4665-9900-20d91cd8c806-maas-api-service-ca\") pod \"b2cdaecf-78a7-4665-9900-20d91cd8c806\" (UID: \"b2cdaecf-78a7-4665-9900-20d91cd8c806\") "
Nov 03 20:16:02.025117 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.025097    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256aebcd-7912-4b33-817e-53b38c027027-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/256aebcd-7912-4b33-817e-53b38c027027-maas-api-service-ca\") pod \"256aebcd-7912-4b33-817e-53b38c027027\" (UID: \"256aebcd-7912-4b33-817e-53b38c027027\") "
Nov 03 20:16:02.025600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.025570    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256aebcd-7912-4b33-817e-53b38c027027-maas-api-service-ca" pod "256aebcd-7912-4b33-817e-53b38c027027" (UID: "256aebcd-7912-4b33-817e-53b38c027027"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:02.025705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.025324    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf21193-3ee2-451e-b37b-6b4a99a08c35-maas-api-service-ca" pod "2bf21193-3ee2-451e-b37b-6b4a99a08c35" (UID: "2bf21193-3ee2-451e-b37b-6b4a99a08c35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:02.025705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.025618    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2cdaecf-78a7-4665-9900-20d91cd8c806-maas-api-service-ca" pod "b2cdaecf-78a7-4665-9900-20d91cd8c806" (UID: "b2cdaecf-78a7-4665-9900-20d91cd8c806"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:02.026092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.026064    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/256aebcd-7912-4b33-817e-53b38c027027-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:02.026194 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.026101    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf21193-3ee2-451e-b37b-6b4a99a08c35-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:02.026194 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.026116    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2cdaecf-78a7-4665-9900-20d91cd8c806-maas-api-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:02.028677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.028611    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256aebcd-7912-4b33-817e-53b38c027027-kube-api-access-4npxw" pod "256aebcd-7912-4b33-817e-53b38c027027" (UID: "256aebcd-7912-4b33-817e-53b38c027027"). InnerVolumeSpecName "kube-api-access-4npxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:02.028868 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.028844    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cdaecf-78a7-4665-9900-20d91cd8c806-kube-api-access-2tsf7" pod "b2cdaecf-78a7-4665-9900-20d91cd8c806" (UID: "b2cdaecf-78a7-4665-9900-20d91cd8c806"). InnerVolumeSpecName "kube-api-access-2tsf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:02.029160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.029137    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf21193-3ee2-451e-b37b-6b4a99a08c35-kube-api-access-ptmxt" pod "2bf21193-3ee2-451e-b37b-6b4a99a08c35" (UID: "2bf21193-3ee2-451e-b37b-6b4a99a08c35"). InnerVolumeSpecName "kube-api-access-ptmxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:02.127891 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.127850    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptmxt\" (UniqueName: \"kubernetes.io/projected/2bf21193-3ee2-451e-b37b-6b4a99a08c35-kube-api-access-ptmxt\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:02.127891 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.127884    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tsf7\" (UniqueName: \"kubernetes.io/projected/b2cdaecf-78a7-4665-9900-20d91cd8c806-kube-api-access-2tsf7\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:02.127891 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.127898    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4npxw\" (UniqueName: \"kubernetes.io/projected/256aebcd-7912-4b33-817e-53b38c027027-kube-api-access-4npxw\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:02.450377 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.450349    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"
Nov 03 20:16:02.450894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.450380    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n" event={"ID":"2bf21193-3ee2-451e-b37b-6b4a99a08c35","Type":"ContainerDied","Data":"3ba943ef1ccfef17a60dd03f364a82f2fe134f15e6605e327ffb1c44a616e770"}
Nov 03 20:16:02.450894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.450478    5600 scope.go:122] "RemoveContainer" containerID="afeaa2bd72257435c8e9cade737f69f9aa6c275e4d60e548262b693e9dc9d5ba"
Nov 03 20:16:02.452110 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.452089    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"
Nov 03 20:16:02.452237 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.452103    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn" event={"ID":"b2cdaecf-78a7-4665-9900-20d91cd8c806","Type":"ContainerDied","Data":"754de1847a9867d348ebb44ee0f771bd041477e33abacbbb8aac400e4a3b9f2e"}
Nov 03 20:16:02.453721 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.453702    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"
Nov 03 20:16:02.453845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.453759    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp" event={"ID":"256aebcd-7912-4b33-817e-53b38c027027","Type":"ContainerDied","Data":"165485b40c1cfc6525eae1b8a75c13c8f4bf3f1ac4c110e64b8e63b026308c58"}
Nov 03 20:16:02.463519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.463496    5600 scope.go:122] "RemoveContainer" containerID="da66d72560a5da9ad20918f5a282bea9e53d3690697125c6b2758f138603af07"
Nov 03 20:16:02.474286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.474265    5600 scope.go:122] "RemoveContainer" containerID="ee3de33aa3bfd7487287b80ad668e5c683829bc4f9b8e324745f16c25294f496"
Nov 03 20:16:02.490326 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.490303    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"]
Nov 03 20:16:02.497271 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.497246    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49fb2b-29895615-fgv9n"]
Nov 03 20:16:02.509033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.508979    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"]
Nov 03 20:16:02.516648 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.516615    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0ffd89-29895615-xxmxp"]
Nov 03 20:16:02.527112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.527084    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"]
Nov 03 20:16:02.530916 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:02.530893    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f817b0-29895615-j2zsn"]
Nov 03 20:16:03.624310 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:03.624275    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256aebcd-7912-4b33-817e-53b38c027027" path="/var/lib/kubelet/pods/256aebcd-7912-4b33-817e-53b38c027027/volumes"
Nov 03 20:16:03.624865 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:03.624847    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf21193-3ee2-451e-b37b-6b4a99a08c35" path="/var/lib/kubelet/pods/2bf21193-3ee2-451e-b37b-6b4a99a08c35/volumes"
Nov 03 20:16:03.625251 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:03.625238    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2cdaecf-78a7-4665-9900-20d91cd8c806" path="/var/lib/kubelet/pods/b2cdaecf-78a7-4665-9900-20d91cd8c806/volumes"
Nov 03 20:16:28.810319 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:16:28.810225    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:16:41.415086 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:41.414583    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-28w6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:16:41.415086 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:16:41.414656    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:58.802742 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:16:58.802701    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:17:09.353743 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:09.353706    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"]
Nov 03 20:17:09.354331 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:09.353941    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy" containerID="cri-o://f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4" gracePeriod=30
Nov 03 20:17:14.698421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.698386    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:17:14.739085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.739042    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"]
Nov 03 20:17:14.740394 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.740363    5600 cpu_manager.go:414] "containerMap: removing container" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerName="cleanup"
Nov 03 20:17:14.740579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.740564    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerName="cleanup"
Nov 03 20:17:14.740671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.740662    5600 cpu_manager.go:414] "containerMap: removing container" podUID="256aebcd-7912-4b33-817e-53b38c027027" containerName="cleanup"
Nov 03 20:17:14.740742 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.740733    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256aebcd-7912-4b33-817e-53b38c027027" containerName="cleanup"
Nov 03 20:17:14.740828 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.740819    5600 cpu_manager.go:414] "containerMap: removing container" podUID="2bf21193-3ee2-451e-b37b-6b4a99a08c35" containerName="cleanup"
Nov 03 20:17:14.740956 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.740894    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf21193-3ee2-451e-b37b-6b4a99a08c35" containerName="cleanup"
Nov 03 20:17:14.741059 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.741050    5600 cpu_manager.go:414] "containerMap: removing container" podUID="2bf21193-3ee2-451e-b37b-6b4a99a08c35" containerName="cleanup"
Nov 03 20:17:14.741133 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.741124    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf21193-3ee2-451e-b37b-6b4a99a08c35" containerName="cleanup"
Nov 03 20:17:14.741378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.741365    5600 cpu_manager.go:414] "containerMap: removing container" podUID="256aebcd-7912-4b33-817e-53b38c027027" containerName="cleanup"
Nov 03 20:17:14.743321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.743301    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256aebcd-7912-4b33-817e-53b38c027027" containerName="cleanup"
Nov 03 20:17:14.743475 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.743461    5600 cpu_manager.go:414] "containerMap: removing container" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy"
Nov 03 20:17:14.746628 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.746567    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy"
Nov 03 20:17:14.746796 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.746778    5600 cpu_manager.go:414] "containerMap: removing container" podUID="b2cdaecf-78a7-4665-9900-20d91cd8c806" containerName="cleanup"
Nov 03 20:17:14.746894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.746882    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cdaecf-78a7-4665-9900-20d91cd8c806" containerName="cleanup"
Nov 03 20:17:14.747008 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.746996    5600 cpu_manager.go:414] "containerMap: removing container" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerName="cleanup"
Nov 03 20:17:14.747095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.747083    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerName="cleanup"
Nov 03 20:17:14.747372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.747356    5600 cpu_manager.go:414] "containerMap: removing container" podUID="b2cdaecf-78a7-4665-9900-20d91cd8c806" containerName="cleanup"
Nov 03 20:17:14.747507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.747495    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cdaecf-78a7-4665-9900-20d91cd8c806" containerName="cleanup"
Nov 03 20:17:14.747825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.747799    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf21193-3ee2-451e-b37b-6b4a99a08c35" containerName="cleanup"
Nov 03 20:17:14.747951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.747938    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf21193-3ee2-451e-b37b-6b4a99a08c35" containerName="cleanup"
Nov 03 20:17:14.748046 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.748036    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="256aebcd-7912-4b33-817e-53b38c027027" containerName="cleanup"
Nov 03 20:17:14.748130 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.748120    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="460ffe79-e497-4547-8d0e-9db401527282" containerName="istio-proxy"
Nov 03 20:17:14.748204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.748195    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cdaecf-78a7-4665-9900-20d91cd8c806" containerName="cleanup"
Nov 03 20:17:14.748277 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.748268    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerName="cleanup"
Nov 03 20:17:14.748353 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.748345    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerName="cleanup"
Nov 03 20:17:14.748452 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.748424    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cdaecf-78a7-4665-9900-20d91cd8c806" containerName="cleanup"
Nov 03 20:17:14.757746 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.757726    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.767504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.762886    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2c6c50-openshift-default-dockercfg-4phd2\""
Nov 03 20:17:14.767504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.763987    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"]
Nov 03 20:17:14.847286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.847188    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-kube-api-access-nd86m\" (UniqueName: \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-kube-api-access-nd86m\") pod \"460ffe79-e497-4547-8d0e-9db401527282\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") "
Nov 03 20:17:14.847530 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.847420    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-istio-token\" (UniqueName: \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-istio-token\") pod \"460ffe79-e497-4547-8d0e-9db401527282\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") "
Nov 03 20:17:14.847615 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.847585    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-envoy\") pod \"460ffe79-e497-4547-8d0e-9db401527282\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") "
Nov 03 20:17:14.847696 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.847676    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-credential-socket\") pod \"460ffe79-e497-4547-8d0e-9db401527282\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") "
Nov 03 20:17:14.847784 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.847732    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/460ffe79-e497-4547-8d0e-9db401527282-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460ffe79-e497-4547-8d0e-9db401527282-istiod-ca-cert\") pod \"460ffe79-e497-4547-8d0e-9db401527282\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") "
Nov 03 20:17:14.847784 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.847774    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-certs\") pod \"460ffe79-e497-4547-8d0e-9db401527282\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") "
Nov 03 20:17:14.847904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.847810    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/460ffe79-e497-4547-8d0e-9db401527282-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460ffe79-e497-4547-8d0e-9db401527282-istio-podinfo\") pod \"460ffe79-e497-4547-8d0e-9db401527282\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") "
Nov 03 20:17:14.847904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.847888    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-socket\") pod \"460ffe79-e497-4547-8d0e-9db401527282\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") "
Nov 03 20:17:14.848010 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.847915    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-data\") pod \"460ffe79-e497-4547-8d0e-9db401527282\" (UID: \"460ffe79-e497-4547-8d0e-9db401527282\") "
Nov 03 20:17:14.848707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.848182    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-certs" pod "460ffe79-e497-4547-8d0e-9db401527282" (UID: "460ffe79-e497-4547-8d0e-9db401527282"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:14.848707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.848334    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.848707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.848619    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.848707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.848354    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-credential-socket" pod "460ffe79-e497-4547-8d0e-9db401527282" (UID: "460ffe79-e497-4547-8d0e-9db401527282"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:14.848707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.848598    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-socket" pod "460ffe79-e497-4547-8d0e-9db401527282" (UID: "460ffe79-e497-4547-8d0e-9db401527282"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:14.848707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.848616    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460ffe79-e497-4547-8d0e-9db401527282-istiod-ca-cert" pod "460ffe79-e497-4547-8d0e-9db401527282" (UID: "460ffe79-e497-4547-8d0e-9db401527282"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:14.848707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.848655    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-data" pod "460ffe79-e497-4547-8d0e-9db401527282" (UID: "460ffe79-e497-4547-8d0e-9db401527282"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:14.849137 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.848814    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.849137 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.848862    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.849137 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.848892    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.849137 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.848919    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.849137 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.849012    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.849388 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.849223    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.849388 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.849297    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2gfp\" (UniqueName: \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-kube-api-access-d2gfp\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.849723 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.849543    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.849723 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.849644    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/460ffe79-e497-4547-8d0e-9db401527282-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.849723 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.849663    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.849723 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.849689    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.849723 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.849703    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.850907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.850852    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-istio-token" pod "460ffe79-e497-4547-8d0e-9db401527282" (UID: "460ffe79-e497-4547-8d0e-9db401527282"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:14.851697 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.851672    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-kube-api-access-nd86m" pod "460ffe79-e497-4547-8d0e-9db401527282" (UID: "460ffe79-e497-4547-8d0e-9db401527282"). InnerVolumeSpecName "kube-api-access-nd86m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:14.851801 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.851759    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/460ffe79-e497-4547-8d0e-9db401527282-istio-podinfo" pod "460ffe79-e497-4547-8d0e-9db401527282" (UID: "460ffe79-e497-4547-8d0e-9db401527282"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:17:14.852246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.852219    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-envoy" pod "460ffe79-e497-4547-8d0e-9db401527282" (UID: "460ffe79-e497-4547-8d0e-9db401527282"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:14.951275 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951182    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.951275 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951226    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2gfp\" (UniqueName: \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-kube-api-access-d2gfp\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.951508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951292    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.951508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951355    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.951508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951457    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.951508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951495    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.951659 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951519    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.951659 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951544    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.951659 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951619    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.951762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951693    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.951809 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951761    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.951938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951901    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.951938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951927    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/460ffe79-e497-4547-8d0e-9db401527282-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.952055 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951944    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/460ffe79-e497-4547-8d0e-9db401527282-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.952055 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951960    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd86m\" (UniqueName: \"kubernetes.io/projected/460ffe79-e497-4547-8d0e-9db401527282-kube-api-access-nd86m\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.952055 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.951926    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.952055 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.952033    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.952423 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.952282    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.953993 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.953968    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.954319 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.954300    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.959284 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.959252    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2gfp\" (UniqueName: \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-kube-api-access-d2gfp\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:14.959577 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:14.959560    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:15.087452 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.087404    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:15.242539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.242398    5600 generic.go:356] "Generic (PLEG): container finished" podID="460ffe79-e497-4547-8d0e-9db401527282" containerID="f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4" exitCode=0
Nov 03 20:17:15.242539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.242473    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" event={"ID":"460ffe79-e497-4547-8d0e-9db401527282","Type":"ContainerDied","Data":"f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4"}
Nov 03 20:17:15.242539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.242524    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"
Nov 03 20:17:15.242825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.242545    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d" event={"ID":"460ffe79-e497-4547-8d0e-9db401527282","Type":"ContainerDied","Data":"050cb464ad0992ef6ddb7aa429b3abf4757669fb6c913b5ebad497eb8943f272"}
Nov 03 20:17:15.242825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.242570    5600 scope.go:122] "RemoveContainer" containerID="f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4"
Nov 03 20:17:15.254876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.254855    5600 scope.go:122] "RemoveContainer" containerID="f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4"
Nov 03 20:17:15.255201 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:17:15.255177    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4\": container with ID starting with f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4 not found: ID does not exist" containerID="f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4"
Nov 03 20:17:15.255290 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.255214    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4"} err="failed to get container status \"f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4\": rpc error: code = NotFound desc = could not find container \"f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4\": container with ID starting with f72ecdf57b65cc1ab1a7289079d240101dae7aed7eb8ba4f7fc1154e7b6032e4 not found: ID does not exist"
Nov 03 20:17:15.269849 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.269820    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"]
Nov 03 20:17:15.275356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.275329    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-n447d"]
Nov 03 20:17:15.384480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.384401    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"]
Nov 03 20:17:15.385013 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:17:15.384982    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc0b3b1_a009_4a94_a0e7_77b7683548e7.slice/crio-003e246b735d68c4002f2fa5a6ac78bebba650a300962610e9d9f06aea671dd4 WatchSource:0}: Error finding container 003e246b735d68c4002f2fa5a6ac78bebba650a300962610e9d9f06aea671dd4: Status 404 returned error can't find the container with id 003e246b735d68c4002f2fa5a6ac78bebba650a300962610e9d9f06aea671dd4
Nov 03 20:17:15.388827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.388786    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:17:15.388988 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.388883    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:17:15.388988 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.388926    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:17:15.621546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:15.621510    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="460ffe79-e497-4547-8d0e-9db401527282" path="/var/lib/kubelet/pods/460ffe79-e497-4547-8d0e-9db401527282/volumes"
Nov 03 20:17:16.253191 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:16.253151    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" event={"ID":"5bc0b3b1-a009-4a94-a0e7-77b7683548e7","Type":"ContainerStarted","Data":"7cb94c8dcd5c53ad2c018d96ee6671adc11d73c3a8d217e1ed2fc4b6934b925b"}
Nov 03 20:17:16.253656 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:16.253255    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" event={"ID":"5bc0b3b1-a009-4a94-a0e7-77b7683548e7","Type":"ContainerStarted","Data":"003e246b735d68c4002f2fa5a6ac78bebba650a300962610e9d9f06aea671dd4"}
Nov 03 20:17:16.276365 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:16.276299    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" podStartSLOduration=55.2762807 podStartE2EDuration="55.2762807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:16.272984901 +0000 UTC m=+2218.785806697" watchObservedRunningTime="2026-11-03 20:17:16.2762807 +0000 UTC m=+2218.789102483"
Nov 03 20:17:17.096460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:17.089720    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:17.096460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:17.089847    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 03 20:17:17.096460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:17.089893    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" podUID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 03 20:17:18.096461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:18.094511    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 03 20:17:18.096461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:18.094582    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" podUID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 03 20:17:19.094477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:19.089848    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 03 20:17:19.094477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:19.089922    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" podUID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 03 20:17:20.101471 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:20.097897    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 03 20:17:20.101471 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:20.097961    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" podUID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 03 20:17:21.101457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:21.101189    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 03 20:17:21.101457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:21.101259    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" podUID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 03 20:17:22.093292 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:22.093254    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:22.093812 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:22.093794    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:22.094529 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:22.094501    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:17:23.961328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:23.961290    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"]
Nov 03 20:17:23.961934 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:23.961517    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerName="istio-proxy" containerID="cri-o://94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6" gracePeriod=30
Nov 03 20:17:28.798425 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:17:28.797398    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:17:29.304497 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.304468    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:17:29.338346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.338254    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"]
Nov 03 20:17:29.339456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.339409    5600 cpu_manager.go:414] "containerMap: removing container" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerName="istio-proxy"
Nov 03 20:17:29.339456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.339452    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerName="istio-proxy"
Nov 03 20:17:29.340047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.339482    5600 cpu_manager.go:414] "containerMap: removing container" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerName="cleanup"
Nov 03 20:17:29.340047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.339490    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerName="cleanup"
Nov 03 20:17:29.340047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.339685    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerName="istio-proxy"
Nov 03 20:17:29.340047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.339712    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="256aebcd-7912-4b33-817e-53b38c027027" containerName="cleanup"
Nov 03 20:17:29.340047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.339725    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb78ddb-9de0-4f4b-afe7-f5ae8721ebcc" containerName="cleanup"
Nov 03 20:17:29.344019 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.343983    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.361688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.361662    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"]
Nov 03 20:17:29.398084 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.398050    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-envoy\") pod \"29cfda3e-9584-49b9-a9b9-4824e6efa782\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") "
Nov 03 20:17:29.398428 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.398406    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29cfda3e-9584-49b9-a9b9-4824e6efa782-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29cfda3e-9584-49b9-a9b9-4824e6efa782-istiod-ca-cert\") pod \"29cfda3e-9584-49b9-a9b9-4824e6efa782\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") "
Nov 03 20:17:29.398733 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.398713    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-data\") pod \"29cfda3e-9584-49b9-a9b9-4824e6efa782\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") "
Nov 03 20:17:29.398915 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.398897    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-token\" (UniqueName: \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-token\") pod \"29cfda3e-9584-49b9-a9b9-4824e6efa782\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") "
Nov 03 20:17:29.399092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.399077    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-certs\") pod \"29cfda3e-9584-49b9-a9b9-4824e6efa782\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") "
Nov 03 20:17:29.399294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.399276    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-socket\") pod \"29cfda3e-9584-49b9-a9b9-4824e6efa782\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") "
Nov 03 20:17:29.399421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.399405    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-credential-socket\") pod \"29cfda3e-9584-49b9-a9b9-4824e6efa782\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") "
Nov 03 20:17:29.399599 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.399582    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-podinfo\") pod \"29cfda3e-9584-49b9-a9b9-4824e6efa782\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") "
Nov 03 20:17:29.399766 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.399748    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-kube-api-access-jqr65\" (UniqueName: \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-kube-api-access-jqr65\") pod \"29cfda3e-9584-49b9-a9b9-4824e6efa782\" (UID: \"29cfda3e-9584-49b9-a9b9-4824e6efa782\") "
Nov 03 20:17:29.400120 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.398958    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29cfda3e-9584-49b9-a9b9-4824e6efa782-istiod-ca-cert" pod "29cfda3e-9584-49b9-a9b9-4824e6efa782" (UID: "29cfda3e-9584-49b9-a9b9-4824e6efa782"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:29.400211 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.399194    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-data" pod "29cfda3e-9584-49b9-a9b9-4824e6efa782" (UID: "29cfda3e-9584-49b9-a9b9-4824e6efa782"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.400211 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.399803    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-credential-socket" pod "29cfda3e-9584-49b9-a9b9-4824e6efa782" (UID: "29cfda3e-9584-49b9-a9b9-4824e6efa782"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.400211 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.400086    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-socket" pod "29cfda3e-9584-49b9-a9b9-4824e6efa782" (UID: "29cfda3e-9584-49b9-a9b9-4824e6efa782"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.400623 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.400599    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89tsj\" (UniqueName: \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-kube-api-access-89tsj\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.400829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.400810    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.401232 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.401213    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.401400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.401349    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.401570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.401555    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e2387a-d83a-4df4-a7c9-781356645f2b-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.401746 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.401727    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.401899 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.401883    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.402112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.402094    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.402270 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.400644    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-certs" pod "29cfda3e-9584-49b9-a9b9-4824e6efa782" (UID: "29cfda3e-9584-49b9-a9b9-4824e6efa782"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.402270 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.402252    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.402626 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.402607    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.402740 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.402728    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.402822 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.402809    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.402919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.402906    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29cfda3e-9584-49b9-a9b9-4824e6efa782-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.403008 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.402997    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.403483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.403452    5600 generic.go:356] "Generic (PLEG): container finished" podID="29cfda3e-9584-49b9-a9b9-4824e6efa782" containerID="94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6" exitCode=0
Nov 03 20:17:29.403573 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.403546    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" event={"ID":"29cfda3e-9584-49b9-a9b9-4824e6efa782","Type":"ContainerDied","Data":"94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6"}
Nov 03 20:17:29.403642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.403598    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm" event={"ID":"29cfda3e-9584-49b9-a9b9-4824e6efa782","Type":"ContainerDied","Data":"aa2918ca1312c283b096dcc789c85713a787ee8bf5a22f4bc4a1cbcf0d925ea0"}
Nov 03 20:17:29.403642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.403630    5600 scope.go:122] "RemoveContainer" containerID="94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6"
Nov 03 20:17:29.403804 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.403788    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"
Nov 03 20:17:29.409266 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.409201    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-podinfo" pod "29cfda3e-9584-49b9-a9b9-4824e6efa782" (UID: "29cfda3e-9584-49b9-a9b9-4824e6efa782"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:17:29.409545 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.409500    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-kube-api-access-jqr65" pod "29cfda3e-9584-49b9-a9b9-4824e6efa782" (UID: "29cfda3e-9584-49b9-a9b9-4824e6efa782"). InnerVolumeSpecName "kube-api-access-jqr65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:29.412937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.412901    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-envoy" pod "29cfda3e-9584-49b9-a9b9-4824e6efa782" (UID: "29cfda3e-9584-49b9-a9b9-4824e6efa782"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.414377 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.414348    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-token" pod "29cfda3e-9584-49b9-a9b9-4824e6efa782" (UID: "29cfda3e-9584-49b9-a9b9-4824e6efa782"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:29.454663 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.454545    5600 scope.go:122] "RemoveContainer" containerID="94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6"
Nov 03 20:17:29.454957 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:17:29.454936    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6\": container with ID starting with 94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6 not found: ID does not exist" containerID="94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6"
Nov 03 20:17:29.455036 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.454968    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6"} err="failed to get container status \"94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6\": rpc error: code = NotFound desc = could not find container \"94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6\": container with ID starting with 94875233fd64b0e3b7f16949e9005f254fc4e1ada10aaa20fb7b473f694c16f6 not found: ID does not exist"
Nov 03 20:17:29.514680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.514638    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.515049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.515027    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.515197 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.515181    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.515325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.515311    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e2387a-d83a-4df4-a7c9-781356645f2b-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.515468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.515451    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.515649 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.515633    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.515850 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.515835    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.515994 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.515976    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.516223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.516207    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89tsj\" (UniqueName: \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-kube-api-access-89tsj\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.516552 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.516534    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.516681 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.516668    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqr65\" (UniqueName: \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-kube-api-access-jqr65\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.516767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.516756    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.533732 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.533707    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29cfda3e-9584-49b9-a9b9-4824e6efa782-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.536423 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.524919    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e2387a-d83a-4df4-a7c9-781356645f2b-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.537382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.537354    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.543201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.543173    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.543798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.543766    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.543943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.543900    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.544043 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.544009    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.544459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.544418    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.546041 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.546016    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.546982 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.546962    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89tsj\" (UniqueName: \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-kube-api-access-89tsj\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.669773 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.669745    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:29.732630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.732591    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"]
Nov 03 20:17:29.744855 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.743645    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-qqwhm"]
Nov 03 20:17:29.976602 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:17:29.976572    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e2387a_d83a_4df4_a7c9_781356645f2b.slice/crio-1d8aec9930bde9915c3101aad6fff35db69f9c38d258f7153e10b1ff964f203a WatchSource:0}: Error finding container 1d8aec9930bde9915c3101aad6fff35db69f9c38d258f7153e10b1ff964f203a: Status 404 returned error can't find the container with id 1d8aec9930bde9915c3101aad6fff35db69f9c38d258f7153e10b1ff964f203a
Nov 03 20:17:29.978278 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.978245    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"]
Nov 03 20:17:29.980170 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.980135    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:17:29.980269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.980226    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:17:29.980269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:29.980265    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:17:30.418199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:30.418141    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" event={"ID":"10e2387a-d83a-4df4-a7c9-781356645f2b","Type":"ContainerStarted","Data":"40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398"}
Nov 03 20:17:30.418524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:30.418243    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" event={"ID":"10e2387a-d83a-4df4-a7c9-781356645f2b","Type":"ContainerStarted","Data":"1d8aec9930bde9915c3101aad6fff35db69f9c38d258f7153e10b1ff964f203a"}
Nov 03 20:17:30.459391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:30.454338    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" podStartSLOduration=69.454316148 podStartE2EDuration="1m9.454316148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:30.448127083 +0000 UTC m=+2232.960948866" watchObservedRunningTime="2026-11-03 20:17:30.454316148 +0000 UTC m=+2232.967137932"
Nov 03 20:17:30.671774 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:30.671679    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:31.620821 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:31.620783    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29cfda3e-9584-49b9-a9b9-4824e6efa782" path="/var/lib/kubelet/pods/29cfda3e-9584-49b9-a9b9-4824e6efa782/volumes"
Nov 03 20:17:31.672880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:31.672838    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 20:17:31.673057 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:31.672895    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" podUID="10e2387a-d83a-4df4-a7c9-781356645f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 20:17:32.670281 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:32.670249    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 20:17:32.670854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:32.670307    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" podUID="10e2387a-d83a-4df4-a7c9-781356645f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 20:17:33.670116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:33.670080    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 20:17:33.670342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:33.670141    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" podUID="10e2387a-d83a-4df4-a7c9-781356645f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 20:17:34.238489 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.235507    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"]
Nov 03 20:17:34.240946 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.240916    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:17:34.241112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.241071    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"]
Nov 03 20:17:34.252137 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.252110    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-46c34c\""
Nov 03 20:17:34.380495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.377206    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m64dx\" (UniqueName: \"kubernetes.io/projected/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-kube-api-access-m64dx\") pod \"payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7\" (UID: \"441b3899-e37a-4c0a-90e4-8cbf1ece2c65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:17:34.380495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.377664    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7\" (UID: \"441b3899-e37a-4c0a-90e4-8cbf1ece2c65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:17:34.486574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.478916    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7\" (UID: \"441b3899-e37a-4c0a-90e4-8cbf1ece2c65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:17:34.486574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.479041    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m64dx\" (UniqueName: \"kubernetes.io/projected/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-kube-api-access-m64dx\") pod \"payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7\" (UID: \"441b3899-e37a-4c0a-90e4-8cbf1ece2c65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:17:34.486574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.480407    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7\" (UID: \"441b3899-e37a-4c0a-90e4-8cbf1ece2c65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:17:34.505602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.501556    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m64dx\" (UniqueName: \"kubernetes.io/projected/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-kube-api-access-m64dx\") pod \"payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7\" (UID: \"441b3899-e37a-4c0a-90e4-8cbf1ece2c65\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:17:34.574459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.564940    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:17:34.680508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.672898    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 20:17:34.680508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:34.672960    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" podUID="10e2387a-d83a-4df4-a7c9-781356645f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 20:17:35.049141 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:35.049116    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"]
Nov 03 20:17:35.520076 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:35.520027    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7" event={"ID":"441b3899-e37a-4c0a-90e4-8cbf1ece2c65","Type":"ContainerStarted","Data":"227c7f9079dc2c4381767925d75ea019f246220d1bf7596e03e3b802a2377637"}
Nov 03 20:17:35.520306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:35.520092    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7" event={"ID":"441b3899-e37a-4c0a-90e4-8cbf1ece2c65","Type":"ContainerStarted","Data":"27fe9345bd5212d34cb090dde1ea39e289295fc5eec4ac0a28f71805be7da3a8"}
Nov 03 20:17:35.520306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:35.520268    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:17:35.541454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:35.541377    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7" podStartSLOduration=1.541363807 podStartE2EDuration="1.541363807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:35.538167696 +0000 UTC m=+2238.050989473" watchObservedRunningTime="2026-11-03 20:17:35.541363807 +0000 UTC m=+2238.054185588"
Nov 03 20:17:35.670716 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:35.670679    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 20:17:35.670897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:35.670740    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" podUID="10e2387a-d83a-4df4-a7c9-781356645f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 20:17:36.676301 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:36.676273    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:36.676737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:36.676670    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:36.677531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:36.677508    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:17:46.534818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:46.534779    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:17:53.529778 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:53.529741    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"]
Nov 03 20:17:53.530286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:53.529993    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" containerName="istio-proxy" containerID="cri-o://0806e50cf74b495556ddf4ba64154dc2838425148861555fcdf10101b0f473fd" gracePeriod=30
Nov 03 20:17:58.801584 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:17:58.801421    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:17:58.834825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.834790    5600 generic.go:356] "Generic (PLEG): container finished" podID="123af875-2394-48b6-baca-ef5056d5a6cf" containerID="0806e50cf74b495556ddf4ba64154dc2838425148861555fcdf10101b0f473fd" exitCode=0
Nov 03 20:17:58.834967 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.834862    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" event={"ID":"123af875-2394-48b6-baca-ef5056d5a6cf","Type":"ContainerDied","Data":"0806e50cf74b495556ddf4ba64154dc2838425148861555fcdf10101b0f473fd"}
Nov 03 20:17:58.834967 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.834927    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd" event={"ID":"123af875-2394-48b6-baca-ef5056d5a6cf","Type":"ContainerDied","Data":"a2798d7ac2ec96c69aa01fcbc761a261ae1207596a5a68e821e1e540c5043bc1"}
Nov 03 20:17:58.834967 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.834950    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2798d7ac2ec96c69aa01fcbc761a261ae1207596a5a68e821e1e540c5043bc1"
Nov 03 20:17:58.843974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.843950    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:17:58.881418 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.881384    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"]
Nov 03 20:17:58.882580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.882553    5600 cpu_manager.go:414] "containerMap: removing container" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" containerName="istio-proxy"
Nov 03 20:17:58.882580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.882580    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" containerName="istio-proxy"
Nov 03 20:17:58.882864 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.882847    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" containerName="istio-proxy"
Nov 03 20:17:58.887347 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.887327    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:58.896613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.896586    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"]
Nov 03 20:17:58.906690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.906663    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-46c34c-serving-cert\""
Nov 03 20:17:58.930130 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.930099    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-credential-socket\") pod \"123af875-2394-48b6-baca-ef5056d5a6cf\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") "
Nov 03 20:17:58.930317 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.930159    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123af875-2394-48b6-baca-ef5056d5a6cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123af875-2394-48b6-baca-ef5056d5a6cf-istiod-ca-cert\") pod \"123af875-2394-48b6-baca-ef5056d5a6cf\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") "
Nov 03 20:17:58.930317 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.930252    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-kube-api-access-stfpj\" (UniqueName: \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-kube-api-access-stfpj\") pod \"123af875-2394-48b6-baca-ef5056d5a6cf\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") "
Nov 03 20:17:58.930454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.930327    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-data\") pod \"123af875-2394-48b6-baca-ef5056d5a6cf\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") "
Nov 03 20:17:58.930520 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.930421    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-credential-socket" pod "123af875-2394-48b6-baca-ef5056d5a6cf" (UID: "123af875-2394-48b6-baca-ef5056d5a6cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:58.930520 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.930481    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-istio-token\" (UniqueName: \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-istio-token\") pod \"123af875-2394-48b6-baca-ef5056d5a6cf\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") "
Nov 03 20:17:58.930622 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.930559    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-envoy\") pod \"123af875-2394-48b6-baca-ef5056d5a6cf\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") "
Nov 03 20:17:58.930675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.930646    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-certs\") pod \"123af875-2394-48b6-baca-ef5056d5a6cf\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") "
Nov 03 20:17:58.930769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.930751    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/123af875-2394-48b6-baca-ef5056d5a6cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123af875-2394-48b6-baca-ef5056d5a6cf-istio-podinfo\") pod \"123af875-2394-48b6-baca-ef5056d5a6cf\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") "
Nov 03 20:17:58.932058 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.931198    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-socket\") pod \"123af875-2394-48b6-baca-ef5056d5a6cf\" (UID: \"123af875-2394-48b6-baca-ef5056d5a6cf\") "
Nov 03 20:17:58.932058 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.931632    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-data" pod "123af875-2394-48b6-baca-ef5056d5a6cf" (UID: "123af875-2394-48b6-baca-ef5056d5a6cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:58.932058 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.931824    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-certs" pod "123af875-2394-48b6-baca-ef5056d5a6cf" (UID: "123af875-2394-48b6-baca-ef5056d5a6cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:58.932058 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.931874    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-socket" pod "123af875-2394-48b6-baca-ef5056d5a6cf" (UID: "123af875-2394-48b6-baca-ef5056d5a6cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:58.932058 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.932010    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123af875-2394-48b6-baca-ef5056d5a6cf-istiod-ca-cert" pod "123af875-2394-48b6-baca-ef5056d5a6cf" (UID: "123af875-2394-48b6-baca-ef5056d5a6cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:58.932421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.932234    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:58.932421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.932255    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123af875-2394-48b6-baca-ef5056d5a6cf-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:58.932421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.932270    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:58.932421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.932285    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:58.932421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.932299    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:58.941702 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.941657    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-envoy" pod "123af875-2394-48b6-baca-ef5056d5a6cf" (UID: "123af875-2394-48b6-baca-ef5056d5a6cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:58.941822 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.941769    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-kube-api-access-stfpj" pod "123af875-2394-48b6-baca-ef5056d5a6cf" (UID: "123af875-2394-48b6-baca-ef5056d5a6cf"). InnerVolumeSpecName "kube-api-access-stfpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:58.942083 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.942065    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-istio-token" pod "123af875-2394-48b6-baca-ef5056d5a6cf" (UID: "123af875-2394-48b6-baca-ef5056d5a6cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:58.943633 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:58.943598    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/123af875-2394-48b6-baca-ef5056d5a6cf-istio-podinfo" pod "123af875-2394-48b6-baca-ef5056d5a6cf" (UID: "123af875-2394-48b6-baca-ef5056d5a6cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:17:59.033838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.033805    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-metrics-certs\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.034075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.033942    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-maas-api-tls\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.034220 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.034184    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a243ff65-42fa-4c4c-97a0-4ed640695dee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.034285 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.034254    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz2pd\" (UniqueName: \"kubernetes.io/projected/a243ff65-42fa-4c4c-97a0-4ed640695dee-kube-api-access-qz2pd\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.034414 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.034399    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123af875-2394-48b6-baca-ef5056d5a6cf-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:59.034504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.034420    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stfpj\" (UniqueName: \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-kube-api-access-stfpj\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:59.034504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.034487    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123af875-2394-48b6-baca-ef5056d5a6cf-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:59.034576 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.034502    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123af875-2394-48b6-baca-ef5056d5a6cf-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:59.141507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.138172    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-maas-api-tls\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.141507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.138504    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a243ff65-42fa-4c4c-97a0-4ed640695dee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.141507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.138548    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz2pd\" (UniqueName: \"kubernetes.io/projected/a243ff65-42fa-4c4c-97a0-4ed640695dee-kube-api-access-qz2pd\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.141507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.138742    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-metrics-certs\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.141507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.140177    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a243ff65-42fa-4c4c-97a0-4ed640695dee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.148392 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.145254    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-maas-api-tls\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.153535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.153501    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-metrics-certs\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.157284 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.157251    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz2pd\" (UniqueName: \"kubernetes.io/projected/a243ff65-42fa-4c4c-97a0-4ed640695dee-kube-api-access-qz2pd\") pod \"maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.218910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.218872    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:17:59.512041 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.512009    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"]
Nov 03 20:17:59.513155 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:17:59.513116    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda243ff65_42fa_4c4c_97a0_4ed640695dee.slice/crio-c28628a91ceacc6487d40102f989dd08062446ccafa5935c1b109469acd0a100 WatchSource:0}: Error finding container c28628a91ceacc6487d40102f989dd08062446ccafa5935c1b109469acd0a100: Status 404 returned error can't find the container with id c28628a91ceacc6487d40102f989dd08062446ccafa5935c1b109469acd0a100
Nov 03 20:17:59.846499 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.846342    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2" event={"ID":"a243ff65-42fa-4c4c-97a0-4ed640695dee","Type":"ContainerStarted","Data":"c28628a91ceacc6487d40102f989dd08062446ccafa5935c1b109469acd0a100"}
Nov 03 20:17:59.847042 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.846517    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"
Nov 03 20:17:59.867193 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.867163    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"]
Nov 03 20:17:59.875527 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:17:59.875492    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-b2rtd"]
Nov 03 20:18:00.858943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:00.858899    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2" event={"ID":"a243ff65-42fa-4c4c-97a0-4ed640695dee","Type":"ContainerStarted","Data":"67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b"}
Nov 03 20:18:00.859372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:00.858992    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:18:00.881534 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:00.881473    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2" podStartSLOduration=12.495123587 podStartE2EDuration="12.881456483s" totalImagesPullingTime="386.332896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:00.877699178 +0000 UTC m=+2263.390520957" watchObservedRunningTime="2026-11-03 20:18:00.881456483 +0000 UTC m=+2263.394278262"
Nov 03 20:18:01.622097 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:01.622058    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123af875-2394-48b6-baca-ef5056d5a6cf" path="/var/lib/kubelet/pods/123af875-2394-48b6-baca-ef5056d5a6cf/volumes"
Nov 03 20:18:06.872490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:06.872457    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:18:21.219954 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:21.219916    5600 scope.go:122] "RemoveContainer" containerID="0806e50cf74b495556ddf4ba64154dc2838425148861555fcdf10101b0f473fd"
Nov 03 20:18:28.801382 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:18:28.801336    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:18:38.803247 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:38.803210    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"]
Nov 03 20:18:38.806027 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:38.803460    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerName="istio-proxy" containerID="cri-o://04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5" gracePeriod=30
Nov 03 20:18:44.129308 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.129274    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:18:44.168752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.168714    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"]
Nov 03 20:18:44.169421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.169403    5600 cpu_manager.go:414] "containerMap: removing container" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerName="istio-proxy"
Nov 03 20:18:44.169421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.169419    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerName="istio-proxy"
Nov 03 20:18:44.169617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.169593    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerName="istio-proxy"
Nov 03 20:18:44.173737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.173699    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:18:44.180152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.178249    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0a7343-dockercfg-7jrmp\""
Nov 03 20:18:44.180152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.178337    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0a7343\""
Nov 03 20:18:44.180944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.180907    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"]
Nov 03 20:18:44.285713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.283767    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-kube-api-access-9xtbh\" (UniqueName: \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-kube-api-access-9xtbh\") pod \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") "
Nov 03 20:18:44.285713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.283861    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-podinfo\") pod \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") "
Nov 03 20:18:44.285713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.283918    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-credential-socket\") pod \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") "
Nov 03 20:18:44.285713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.283998    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-socket\") pod \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") "
Nov 03 20:18:44.285713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.284032    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-data\") pod \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") "
Nov 03 20:18:44.285713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.284083    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-certs\") pod \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") "
Nov 03 20:18:44.285713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.284135    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istiod-ca-cert\") pod \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") "
Nov 03 20:18:44.285713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.284212    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-token\" (UniqueName: \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-token\") pod \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") "
Nov 03 20:18:44.285713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.284256    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-envoy\") pod \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\" (UID: \"7ecdfd4b-64e1-4d0a-9114-a78870ae064a\") "
Nov 03 20:18:44.285713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.284711    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c3f058-5ae3-4ad2-be54-af79ec0d4905-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk\" (UID: \"50c3f058-5ae3-4ad2-be54-af79ec0d4905\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:18:44.285713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.284863    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjl4\" (UniqueName: \"kubernetes.io/projected/50c3f058-5ae3-4ad2-be54-af79ec0d4905-kube-api-access-hqjl4\") pod \"payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk\" (UID: \"50c3f058-5ae3-4ad2-be54-af79ec0d4905\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:18:44.287133 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.285863    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-certs" pod "7ecdfd4b-64e1-4d0a-9114-a78870ae064a" (UID: "7ecdfd4b-64e1-4d0a-9114-a78870ae064a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:44.287133 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.286886    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-data" pod "7ecdfd4b-64e1-4d0a-9114-a78870ae064a" (UID: "7ecdfd4b-64e1-4d0a-9114-a78870ae064a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:44.287133 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.287085    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-credential-socket" pod "7ecdfd4b-64e1-4d0a-9114-a78870ae064a" (UID: "7ecdfd4b-64e1-4d0a-9114-a78870ae064a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:44.287617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.287590    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-socket" pod "7ecdfd4b-64e1-4d0a-9114-a78870ae064a" (UID: "7ecdfd4b-64e1-4d0a-9114-a78870ae064a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:44.287714 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.287637    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istiod-ca-cert" pod "7ecdfd4b-64e1-4d0a-9114-a78870ae064a" (UID: "7ecdfd4b-64e1-4d0a-9114-a78870ae064a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:44.289349 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.289326    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-envoy" pod "7ecdfd4b-64e1-4d0a-9114-a78870ae064a" (UID: "7ecdfd4b-64e1-4d0a-9114-a78870ae064a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:44.289542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.289521    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-podinfo" pod "7ecdfd4b-64e1-4d0a-9114-a78870ae064a" (UID: "7ecdfd4b-64e1-4d0a-9114-a78870ae064a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:18:44.290109 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.290086    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-token" pod "7ecdfd4b-64e1-4d0a-9114-a78870ae064a" (UID: "7ecdfd4b-64e1-4d0a-9114-a78870ae064a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:44.290174 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.290123    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-kube-api-access-9xtbh" pod "7ecdfd4b-64e1-4d0a-9114-a78870ae064a" (UID: "7ecdfd4b-64e1-4d0a-9114-a78870ae064a"). InnerVolumeSpecName "kube-api-access-9xtbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:44.383633 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.383600    5600 generic.go:356] "Generic (PLEG): container finished" podID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" containerID="04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5" exitCode=0
Nov 03 20:18:44.383831 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.383669    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"
Nov 03 20:18:44.383894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.383862    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" event={"ID":"7ecdfd4b-64e1-4d0a-9114-a78870ae064a","Type":"ContainerDied","Data":"04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5"}
Nov 03 20:18:44.383948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.383937    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f" event={"ID":"7ecdfd4b-64e1-4d0a-9114-a78870ae064a","Type":"ContainerDied","Data":"c4b22f091dc7b719ba2245eeb01541e7ea22ff9cc2b478f74f9bf18c70561fed"}
Nov 03 20:18:44.384001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.383968    5600 scope.go:122] "RemoveContainer" containerID="04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5"
Nov 03 20:18:44.386722 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.386695    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjl4\" (UniqueName: \"kubernetes.io/projected/50c3f058-5ae3-4ad2-be54-af79ec0d4905-kube-api-access-hqjl4\") pod \"payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk\" (UID: \"50c3f058-5ae3-4ad2-be54-af79ec0d4905\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:18:44.386986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.386967    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c3f058-5ae3-4ad2-be54-af79ec0d4905-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk\" (UID: \"50c3f058-5ae3-4ad2-be54-af79ec0d4905\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:18:44.387078 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.387063    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:44.387140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.387083    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:44.387140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.387098    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:44.387140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.387111    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:44.387140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.387123    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:44.387140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.387137    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:44.387400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.387151    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xtbh\" (UniqueName: \"kubernetes.io/projected/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-kube-api-access-9xtbh\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:44.387400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.387164    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:44.387400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.387177    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecdfd4b-64e1-4d0a-9114-a78870ae064a-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:44.388054 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.388036    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c3f058-5ae3-4ad2-be54-af79ec0d4905-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk\" (UID: \"50c3f058-5ae3-4ad2-be54-af79ec0d4905\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:18:44.395197 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.395092    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjl4\" (UniqueName: \"kubernetes.io/projected/50c3f058-5ae3-4ad2-be54-af79ec0d4905-kube-api-access-hqjl4\") pod \"payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk\" (UID: \"50c3f058-5ae3-4ad2-be54-af79ec0d4905\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:18:44.399524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.399501    5600 scope.go:122] "RemoveContainer" containerID="04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5"
Nov 03 20:18:44.399929 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:18:44.399906    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5\": container with ID starting with 04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5 not found: ID does not exist" containerID="04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5"
Nov 03 20:18:44.399996 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.399941    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5"} err="failed to get container status \"04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5\": rpc error: code = NotFound desc = could not find container \"04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5\": container with ID starting with 04538c3fe683880f823ae44942c5522e254467789a193656cadd65aa6ba478d5 not found: ID does not exist"
Nov 03 20:18:44.415977 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.415925    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"]
Nov 03 20:18:44.422706 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.422675    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkj7f"]
Nov 03 20:18:44.514965 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.514933    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:18:44.829769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:44.829734    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"]
Nov 03 20:18:44.829988 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:18:44.829940    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c3f058_5ae3_4ad2_be54_af79ec0d4905.slice/crio-85f8a0b6b859c92358cc2e54c922f8e683eb381b5739ed67ffe0c7a5edf455c5 WatchSource:0}: Error finding container 85f8a0b6b859c92358cc2e54c922f8e683eb381b5739ed67ffe0c7a5edf455c5: Status 404 returned error can't find the container with id 85f8a0b6b859c92358cc2e54c922f8e683eb381b5739ed67ffe0c7a5edf455c5
Nov 03 20:18:45.395094 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:45.395048    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk" event={"ID":"50c3f058-5ae3-4ad2-be54-af79ec0d4905","Type":"ContainerStarted","Data":"33ec1a50f4d7a98474dec8a91554e4b6bf6d900263ee002189acc5e2357b01ad"}
Nov 03 20:18:45.395605 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:45.395126    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk" event={"ID":"50c3f058-5ae3-4ad2-be54-af79ec0d4905","Type":"ContainerStarted","Data":"85f8a0b6b859c92358cc2e54c922f8e683eb381b5739ed67ffe0c7a5edf455c5"}
Nov 03 20:18:45.395605 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:45.395156    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:18:45.425804 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:45.425742    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk" podStartSLOduration=11.425727143 podStartE2EDuration="11.425727143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:45.420152128 +0000 UTC m=+2307.932973909" watchObservedRunningTime="2026-11-03 20:18:45.425727143 +0000 UTC m=+2307.938548976"
Nov 03 20:18:45.624879 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:45.624658    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ecdfd4b-64e1-4d0a-9114-a78870ae064a" path="/var/lib/kubelet/pods/7ecdfd4b-64e1-4d0a-9114-a78870ae064a/volumes"
Nov 03 20:18:56.387340 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:56.387283    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"]
Nov 03 20:18:56.569460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:18:56.563224    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:18:58.822333 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:18:58.822289    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:19:11.591324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:11.591267    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr" podUID="f9a8989b-01b5-4041-b0b4-63591932d918" containerName="main" containerID="cri-o://4494046154f63eb743c439be9d1f52d438b0517251b42089421277a006a536ee" gracePeriod=45
Nov 03 20:19:11.743671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:11.743630    5600 generic.go:356] "Generic (PLEG): container finished" podID="f9a8989b-01b5-4041-b0b4-63591932d918" containerID="4494046154f63eb743c439be9d1f52d438b0517251b42089421277a006a536ee" exitCode=0
Nov 03 20:19:11.744052 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:11.743952    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr" event={"ID":"f9a8989b-01b5-4041-b0b4-63591932d918","Type":"ContainerDied","Data":"4494046154f63eb743c439be9d1f52d438b0517251b42089421277a006a536ee"}
Nov 03 20:19:11.746497 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:11.746459    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr" podUID="f9a8989b-01b5-4041-b0b4-63591932d918" containerName="main" probeResult="failure" output="Get \"https://10.134.0.117:8000/ready\": dial tcp 10.134.0.117:8000: connect: connection refused"
Nov 03 20:19:11.918132 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:11.918103    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:19:12.005068 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.005025    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-model-cache\") pod \"f9a8989b-01b5-4041-b0b4-63591932d918\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") "
Nov 03 20:19:12.005371 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.005339    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-tmp-dir\") pod \"f9a8989b-01b5-4041-b0b4-63591932d918\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") "
Nov 03 20:19:12.005517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.005391    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-dshm\") pod \"f9a8989b-01b5-4041-b0b4-63591932d918\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") "
Nov 03 20:19:12.005593 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.005550    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a8989b-01b5-4041-b0b4-63591932d918-kube-api-access-s8r9j\" (UniqueName: \"kubernetes.io/projected/f9a8989b-01b5-4041-b0b4-63591932d918-kube-api-access-s8r9j\") pod \"f9a8989b-01b5-4041-b0b4-63591932d918\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") "
Nov 03 20:19:12.006026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.005988    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9a8989b-01b5-4041-b0b4-63591932d918-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a8989b-01b5-4041-b0b4-63591932d918-tls-certs\") pod \"f9a8989b-01b5-4041-b0b4-63591932d918\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") "
Nov 03 20:19:12.006138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.006054    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-home\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-home\") pod \"f9a8989b-01b5-4041-b0b4-63591932d918\" (UID: \"f9a8989b-01b5-4041-b0b4-63591932d918\") "
Nov 03 20:19:12.006228 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.006151    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-tmp-dir" pod "f9a8989b-01b5-4041-b0b4-63591932d918" (UID: "f9a8989b-01b5-4041-b0b4-63591932d918"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:12.006680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.006566    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-home" pod "f9a8989b-01b5-4041-b0b4-63591932d918" (UID: "f9a8989b-01b5-4041-b0b4-63591932d918"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:12.006680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.006653    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-model-cache" pod "f9a8989b-01b5-4041-b0b4-63591932d918" (UID: "f9a8989b-01b5-4041-b0b4-63591932d918"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:12.006908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.006875    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.006908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.006901    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.007092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.006917    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.011869 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.011824    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a8989b-01b5-4041-b0b4-63591932d918-kube-api-access-s8r9j" pod "f9a8989b-01b5-4041-b0b4-63591932d918" (UID: "f9a8989b-01b5-4041-b0b4-63591932d918"). InnerVolumeSpecName "kube-api-access-s8r9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:12.036524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.033069    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-dshm" pod "f9a8989b-01b5-4041-b0b4-63591932d918" (UID: "f9a8989b-01b5-4041-b0b4-63591932d918"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:12.037304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.037271    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a8989b-01b5-4041-b0b4-63591932d918-tls-certs" pod "f9a8989b-01b5-4041-b0b4-63591932d918" (UID: "f9a8989b-01b5-4041-b0b4-63591932d918"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:12.108544 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.108507    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a8989b-01b5-4041-b0b4-63591932d918-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.108544 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.108543    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a8989b-01b5-4041-b0b4-63591932d918-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.108769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.108557    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8r9j\" (UniqueName: \"kubernetes.io/projected/f9a8989b-01b5-4041-b0b4-63591932d918-kube-api-access-s8r9j\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.757070 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.757035    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"
Nov 03 20:19:12.757070 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.757062    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr" event={"ID":"f9a8989b-01b5-4041-b0b4-63591932d918","Type":"ContainerDied","Data":"1aae8c26326d8ec4a8c6168d380e1ff7c28bf5d6c2786453d1925b40a93b6216"}
Nov 03 20:19:12.757602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.757119    5600 scope.go:122] "RemoveContainer" containerID="4494046154f63eb743c439be9d1f52d438b0517251b42089421277a006a536ee"
Nov 03 20:19:12.786488 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.786406    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"]
Nov 03 20:19:12.791513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:12.791482    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f817b0/test-model-f817b0-kserve-8479d856d7-x4vpr"]
Nov 03 20:19:13.618341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:13.618302    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a8989b-01b5-4041-b0b4-63591932d918" path="/var/lib/kubelet/pods/f9a8989b-01b5-4041-b0b4-63591932d918/volumes"
Nov 03 20:19:21.258177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:21.258133    5600 scope.go:122] "RemoveContainer" containerID="9ac7b6d9826f2abc2c7de0e3bb6bb15c2a3f0fe8257a4e56808b083de2d3ff99"
Nov 03 20:19:21.270024 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:21.270000    5600 scope.go:122] "RemoveContainer" containerID="fe936826800b8a8e16ed57ef3e88c99f01b19c8397407835ee3c941684274791"
Nov 03 20:19:24.408007 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:24.407967    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"]
Nov 03 20:19:24.408579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:24.408274    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy" containerID="cri-o://0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48" gracePeriod=30
Nov 03 20:19:28.821143 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:19:28.821032    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:19:29.763536 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.763502    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:19:29.864599 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.864567    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-credential-socket\") pod \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") "
Nov 03 20:19:29.865095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.864631    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-kube-api-access-gmbpg\" (UniqueName: \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-kube-api-access-gmbpg\") pod \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") "
Nov 03 20:19:29.865095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.864751    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istiod-ca-cert\") pod \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") "
Nov 03 20:19:29.865095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.864825    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-token\" (UniqueName: \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-token\") pod \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") "
Nov 03 20:19:29.865095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.864902    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-podinfo\") pod \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") "
Nov 03 20:19:29.865095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.864919    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-credential-socket" pod "7c872495-4d89-43cf-8936-2d2cc87e2fa2" (UID: "7c872495-4d89-43cf-8936-2d2cc87e2fa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.865370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.865327    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istiod-ca-cert" pod "7c872495-4d89-43cf-8936-2d2cc87e2fa2" (UID: "7c872495-4d89-43cf-8936-2d2cc87e2fa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:29.865610 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.865583    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-envoy\") pod \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") "
Nov 03 20:19:29.865762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.865631    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-socket\") pod \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") "
Nov 03 20:19:29.865762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.865656    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-data\") pod \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") "
Nov 03 20:19:29.865762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.865682    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-certs\") pod \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\" (UID: \"7c872495-4d89-43cf-8936-2d2cc87e2fa2\") "
Nov 03 20:19:29.866843 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.866454    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-data" pod "7c872495-4d89-43cf-8936-2d2cc87e2fa2" (UID: "7c872495-4d89-43cf-8936-2d2cc87e2fa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.866843 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.866513    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.866843 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.866534    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.866843 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.866762    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-socket" pod "7c872495-4d89-43cf-8936-2d2cc87e2fa2" (UID: "7c872495-4d89-43cf-8936-2d2cc87e2fa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.866843 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.866820    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-certs" pod "7c872495-4d89-43cf-8936-2d2cc87e2fa2" (UID: "7c872495-4d89-43cf-8936-2d2cc87e2fa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.868514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.868298    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-envoy" pod "7c872495-4d89-43cf-8936-2d2cc87e2fa2" (UID: "7c872495-4d89-43cf-8936-2d2cc87e2fa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.868514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.868367    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-token" pod "7c872495-4d89-43cf-8936-2d2cc87e2fa2" (UID: "7c872495-4d89-43cf-8936-2d2cc87e2fa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:29.868514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.868476    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-kube-api-access-gmbpg" pod "7c872495-4d89-43cf-8936-2d2cc87e2fa2" (UID: "7c872495-4d89-43cf-8936-2d2cc87e2fa2"). InnerVolumeSpecName "kube-api-access-gmbpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:29.868734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.868535    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-podinfo" pod "7c872495-4d89-43cf-8936-2d2cc87e2fa2" (UID: "7c872495-4d89-43cf-8936-2d2cc87e2fa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:29.968307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.968207    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.968307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.968247    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.968307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.968260    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.968307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.968272    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c872495-4d89-43cf-8936-2d2cc87e2fa2-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.968307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.968284    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmbpg\" (UniqueName: \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-kube-api-access-gmbpg\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.968307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.968298    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.968307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:29.968310    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c872495-4d89-43cf-8936-2d2cc87e2fa2-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:30.163932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:30.163898    5600 generic.go:356] "Generic (PLEG): container finished" podID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerID="0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48" exitCode=0
Nov 03 20:19:30.164141 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:30.163947    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"
Nov 03 20:19:30.164141 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:30.163990    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" event={"ID":"7c872495-4d89-43cf-8936-2d2cc87e2fa2","Type":"ContainerDied","Data":"0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48"}
Nov 03 20:19:30.164141 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:30.164050    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq" event={"ID":"7c872495-4d89-43cf-8936-2d2cc87e2fa2","Type":"ContainerDied","Data":"d249c4907304f2aa51f1622465b0053fd902165cdc3ad788f80dbf211e0a7d79"}
Nov 03 20:19:30.164141 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:30.164082    5600 scope.go:122] "RemoveContainer" containerID="0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48"
Nov 03 20:19:30.176489 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:30.176467    5600 scope.go:122] "RemoveContainer" containerID="0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48"
Nov 03 20:19:30.176828 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:19:30.176806    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48\": container with ID starting with 0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48 not found: ID does not exist" containerID="0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48"
Nov 03 20:19:30.176912 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:30.176837    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48"} err="failed to get container status \"0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48\": rpc error: code = NotFound desc = could not find container \"0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48\": container with ID starting with 0882470fe13de6a7ab53bf72bb299ed18d7988a9a6adfb241226660165fddf48 not found: ID does not exist"
Nov 03 20:19:30.192275 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:30.192244    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"]
Nov 03 20:19:30.196452 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:30.196408    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-grkqq"]
Nov 03 20:19:31.620629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:31.620597    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" path="/var/lib/kubelet/pods/7c872495-4d89-43cf-8936-2d2cc87e2fa2/volumes"
Nov 03 20:19:35.036727 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:35.019935    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"]
Nov 03 20:19:35.038272 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:35.038233    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy" containerID="cri-o://c06ae4f0a3d2359b78acee69204dfd186b60138111f2a8c6132f8801ea7f9dba" gracePeriod=30
Nov 03 20:19:35.400187 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:35.400150    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"]
Nov 03 20:19:38.758369 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:38.758330    5600 patch_prober.go:29] interesting pod/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:19:38.758845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:38.758409    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:19:38.888002 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:38.887963    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"]
Nov 03 20:19:38.888233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:38.888202    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy" containerID="cri-o://c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05" gracePeriod=30
Nov 03 20:19:40.533386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.533344    5600 generic.go:356] "Generic (PLEG): container finished" podID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerID="c06ae4f0a3d2359b78acee69204dfd186b60138111f2a8c6132f8801ea7f9dba" exitCode=0
Nov 03 20:19:40.533816 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.533582    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" event={"ID":"aec08d5e-0c0e-459e-85b4-0b80b5b60b35","Type":"ContainerDied","Data":"c06ae4f0a3d2359b78acee69204dfd186b60138111f2a8c6132f8801ea7f9dba"}
Nov 03 20:19:40.533816 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.533643    5600 scope.go:122] "RemoveContainer" containerID="1e37cbfa6f62d5c14de64e1b710182d24fa494a379f0632b2389f04fb5905a15"
Nov 03 20:19:40.941724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.941684    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:19:40.995554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.988937    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-credential-socket\") pod \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") "
Nov 03 20:19:40.995554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.988995    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-token\" (UniqueName: \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-token\") pod \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") "
Nov 03 20:19:40.995554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.989031    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istiod-ca-cert\") pod \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") "
Nov 03 20:19:40.995554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.989063    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-kube-api-access-s76l9\" (UniqueName: \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-kube-api-access-s76l9\") pod \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") "
Nov 03 20:19:40.995554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.989203    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-data\") pod \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") "
Nov 03 20:19:40.995554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.989246    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-socket\") pod \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") "
Nov 03 20:19:40.995554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.989599    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-podinfo\") pod \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") "
Nov 03 20:19:40.995554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.989656    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-certs\") pod \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") "
Nov 03 20:19:40.995554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.989687    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-envoy\") pod \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\" (UID: \"aec08d5e-0c0e-459e-85b4-0b80b5b60b35\") "
Nov 03 20:19:40.995554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.992362    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-credential-socket" pod "aec08d5e-0c0e-459e-85b4-0b80b5b60b35" (UID: "aec08d5e-0c0e-459e-85b4-0b80b5b60b35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:40.995554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.993587    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istiod-ca-cert" pod "aec08d5e-0c0e-459e-85b4-0b80b5b60b35" (UID: "aec08d5e-0c0e-459e-85b4-0b80b5b60b35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:40.996311 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.995571    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-socket" pod "aec08d5e-0c0e-459e-85b4-0b80b5b60b35" (UID: "aec08d5e-0c0e-459e-85b4-0b80b5b60b35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:40.996311 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.995961    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-certs" pod "aec08d5e-0c0e-459e-85b4-0b80b5b60b35" (UID: "aec08d5e-0c0e-459e-85b4-0b80b5b60b35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:40.996311 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.996023    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-data" pod "aec08d5e-0c0e-459e-85b4-0b80b5b60b35" (UID: "aec08d5e-0c0e-459e-85b4-0b80b5b60b35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:40.996986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:40.996957    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-envoy" pod "aec08d5e-0c0e-459e-85b4-0b80b5b60b35" (UID: "aec08d5e-0c0e-459e-85b4-0b80b5b60b35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:41.006503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.006078    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-token" pod "aec08d5e-0c0e-459e-85b4-0b80b5b60b35" (UID: "aec08d5e-0c0e-459e-85b4-0b80b5b60b35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:41.036090 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.036050    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-kube-api-access-s76l9" pod "aec08d5e-0c0e-459e-85b4-0b80b5b60b35" (UID: "aec08d5e-0c0e-459e-85b4-0b80b5b60b35"). InnerVolumeSpecName "kube-api-access-s76l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:41.038270 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.038234    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-podinfo" pod "aec08d5e-0c0e-459e-85b4-0b80b5b60b35" (UID: "aec08d5e-0c0e-459e-85b4-0b80b5b60b35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:41.092060 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.092008    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.092060 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.092053    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.092060 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.092068    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.092502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.092084    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.092502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.092097    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.092502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.092110    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s76l9\" (UniqueName: \"kubernetes.io/projected/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-kube-api-access-s76l9\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.092502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.092124    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.092502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.092137    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.092502 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.092150    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec08d5e-0c0e-459e-85b4-0b80b5b60b35-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.548841 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.548802    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw" event={"ID":"aec08d5e-0c0e-459e-85b4-0b80b5b60b35","Type":"ContainerDied","Data":"fecf13b430e4f6326fb818edba8bb3cfaeafd5d4d368ee4887e384d1472b2b0a"}
Nov 03 20:19:41.549261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.548874    5600 scope.go:122] "RemoveContainer" containerID="c06ae4f0a3d2359b78acee69204dfd186b60138111f2a8c6132f8801ea7f9dba"
Nov 03 20:19:41.549261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.548991    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"
Nov 03 20:19:41.580175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.580138    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"]
Nov 03 20:19:41.585385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.585356    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-43b780-openshift-default-5976c46f47-jx5mw"]
Nov 03 20:19:41.621337 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.621302    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" path="/var/lib/kubelet/pods/aec08d5e-0c0e-459e-85b4-0b80b5b60b35/volumes"
Nov 03 20:19:41.942459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.933969    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"]
Nov 03 20:19:41.942459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:41.934299    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw" podUID="026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" containerName="discovery" containerID="cri-o://48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca" gracePeriod=30
Nov 03 20:19:42.395415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.395381    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:19:42.426023 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.425988    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-dns-cert\") pod \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") "
Nov 03 20:19:42.426253 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.426168    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-token\" (UniqueName: \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-token\") pod \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") "
Nov 03 20:19:42.426253 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.426203    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-cacerts\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-cacerts\") pod \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") "
Nov 03 20:19:42.426385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.426259    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-ca-configmap\") pod \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") "
Nov 03 20:19:42.426385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.426294    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-kube-api-access-6vlvm\" (UniqueName: \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-kube-api-access-6vlvm\") pod \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") "
Nov 03 20:19:42.426565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.426427    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-local-certs\") pod \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") "
Nov 03 20:19:42.426565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.426495    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-kubeconfig\") pod \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\" (UID: \"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25\") "
Nov 03 20:19:42.430868 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.430819    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-kubeconfig" pod "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" (UID: "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:42.436460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.435233    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-ca-configmap" pod "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" (UID: "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:42.438409 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.437358    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-dns-cert" pod "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" (UID: "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:42.438409 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.438225    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-local-certs" pod "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" (UID: "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:42.439635 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.439597    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-cacerts" pod "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" (UID: "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:42.443798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.443768    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-token" pod "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" (UID: "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:42.445149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.445117    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-kube-api-access-6vlvm" pod "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" (UID: "026a1ca8-f2a5-4f87-8adb-b1a8125d0c25"). InnerVolumeSpecName "kube-api-access-6vlvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:42.529225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.529122    5600 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-dns-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:42.529225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.529154    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:42.529225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.529163    5600 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-cacerts\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:42.529225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.529175    5600 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-csr-ca-configmap\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:42.529225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.529191    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vlvm\" (UniqueName: \"kubernetes.io/projected/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-kube-api-access-6vlvm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:42.529225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.529202    5600 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-local-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:42.529225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.529211    5600 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25-istio-kubeconfig\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:42.623457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.623085    5600 generic.go:356] "Generic (PLEG): container finished" podID="026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" containerID="48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca" exitCode=0
Nov 03 20:19:42.623457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.623266    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw" event={"ID":"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25","Type":"ContainerDied","Data":"48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca"}
Nov 03 20:19:42.623457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.623316    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw" event={"ID":"026a1ca8-f2a5-4f87-8adb-b1a8125d0c25","Type":"ContainerDied","Data":"a194982e36dd8585ed9be944f684a0c17ee6409cb945d576897f0780ac273735"}
Nov 03 20:19:42.623457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.623341    5600 scope.go:122] "RemoveContainer" containerID="48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca"
Nov 03 20:19:42.624261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.623500    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"
Nov 03 20:19:42.649206 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.649174    5600 scope.go:122] "RemoveContainer" containerID="48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca"
Nov 03 20:19:42.649846 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:19:42.649822    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca\": container with ID starting with 48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca not found: ID does not exist" containerID="48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca"
Nov 03 20:19:42.650048 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.650017    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca"} err="failed to get container status \"48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca\": rpc error: code = NotFound desc = could not find container \"48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca\": container with ID starting with 48d06e53ebc5241f7671c8cc5e1ee02af9ed0ffca8841286b98a154ee0e658ca not found: ID does not exist"
Nov 03 20:19:42.661545 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.661519    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"]
Nov 03 20:19:42.665193 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:42.665163    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9cmmw"]
Nov 03 20:19:43.620684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:43.620647    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" path="/var/lib/kubelet/pods/026a1ca8-f2a5-4f87-8adb-b1a8125d0c25/volumes"
Nov 03 20:19:44.232010 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.231980    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:19:44.257037 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.257003    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-credential-socket\") pod \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") "
Nov 03 20:19:44.257227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.257078    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-socket\") pod \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") "
Nov 03 20:19:44.257227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.257127    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-certs\") pod \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") "
Nov 03 20:19:44.257227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.257170    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-kube-api-access-psdpc\" (UniqueName: \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-kube-api-access-psdpc\") pod \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") "
Nov 03 20:19:44.257403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.257304    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istiod-ca-cert\") pod \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") "
Nov 03 20:19:44.257403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.257346    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-token\" (UniqueName: \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-token\") pod \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") "
Nov 03 20:19:44.257567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.257416    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-envoy\") pod \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") "
Nov 03 20:19:44.257567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.257473    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-data\") pod \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") "
Nov 03 20:19:44.257668 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.257622    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-podinfo\") pod \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\" (UID: \"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe\") "
Nov 03 20:19:44.260066 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.260005    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-credential-socket" pod "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" (UID: "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.260423 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.260402    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-socket" pod "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" (UID: "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.260929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.260908    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-certs" pod "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" (UID: "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.261528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.261240    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istiod-ca-cert" pod "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" (UID: "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:44.261528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.261499    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-data" pod "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" (UID: "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.266449 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.266396    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-podinfo" pod "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" (UID: "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:44.266747 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.266504    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-kube-api-access-psdpc" pod "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" (UID: "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe"). InnerVolumeSpecName "kube-api-access-psdpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:44.272729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.272635    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-token" pod "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" (UID: "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:44.273705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.273666    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-envoy" pod "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" (UID: "d92a40ff-88c1-4ba0-99cf-47b494fe2ebe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.360189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.360150    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.360189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.360178    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.360189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.360192    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.360455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.360206    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.360455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.360218    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.360455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.360226    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.360455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.360235    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.360455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.360244    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.360455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.360254    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psdpc\" (UniqueName: \"kubernetes.io/projected/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe-kube-api-access-psdpc\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.658672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.658595    5600 generic.go:356] "Generic (PLEG): container finished" podID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerID="c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05" exitCode=0
Nov 03 20:19:44.659001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.658758    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" event={"ID":"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe","Type":"ContainerDied","Data":"c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05"}
Nov 03 20:19:44.659001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.658817    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22" event={"ID":"d92a40ff-88c1-4ba0-99cf-47b494fe2ebe","Type":"ContainerDied","Data":"222c42ada4d72744ee89d131ee691be6122720d3073d85282a4e6c326362bb99"}
Nov 03 20:19:44.659001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.658844    5600 scope.go:122] "RemoveContainer" containerID="c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05"
Nov 03 20:19:44.659001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.658970    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"
Nov 03 20:19:44.675026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.674998    5600 scope.go:122] "RemoveContainer" containerID="c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05"
Nov 03 20:19:44.675749 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:19:44.675706    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05\": container with ID starting with c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05 not found: ID does not exist" containerID="c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05"
Nov 03 20:19:44.675927 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.675894    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05"} err="failed to get container status \"c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05\": rpc error: code = NotFound desc = could not find container \"c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05\": container with ID starting with c53cb7cf393449226180016bb1b8951405ba2f275175cc32d2f84a4358a35b05 not found: ID does not exist"
Nov 03 20:19:44.693260 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.693217    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"]
Nov 03 20:19:44.699498 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:44.699467    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-bmw22"]
Nov 03 20:19:45.621391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:45.621357    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" path="/var/lib/kubelet/pods/d92a40ff-88c1-4ba0-99cf-47b494fe2ebe/volumes"
Nov 03 20:19:48.583601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:48.583257    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"]
Nov 03 20:19:49.649934 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.649905    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"]
Nov 03 20:19:49.650682 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650663    5600 cpu_manager.go:414] "containerMap: removing container" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy"
Nov 03 20:19:49.650682 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650683    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy"
Nov 03 20:19:49.650830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650693    5600 cpu_manager.go:414] "containerMap: removing container" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy"
Nov 03 20:19:49.650830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650698    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy"
Nov 03 20:19:49.650830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650729    5600 cpu_manager.go:414] "containerMap: removing container" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy"
Nov 03 20:19:49.650830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650735    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy"
Nov 03 20:19:49.650830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650755    5600 cpu_manager.go:414] "containerMap: removing container" podUID="026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" containerName="discovery"
Nov 03 20:19:49.650830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650760    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" containerName="discovery"
Nov 03 20:19:49.650830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650768    5600 cpu_manager.go:414] "containerMap: removing container" podUID="f9a8989b-01b5-4041-b0b4-63591932d918" containerName="main"
Nov 03 20:19:49.650830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650773    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a8989b-01b5-4041-b0b4-63591932d918" containerName="main"
Nov 03 20:19:49.651244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650897    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="026a1ca8-f2a5-4f87-8adb-b1a8125d0c25" containerName="discovery"
Nov 03 20:19:49.651244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650905    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a8989b-01b5-4041-b0b4-63591932d918" containerName="main"
Nov 03 20:19:49.651244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650913    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy"
Nov 03 20:19:49.651244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650921    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c872495-4d89-43cf-8936-2d2cc87e2fa2" containerName="istio-proxy"
Nov 03 20:19:49.651244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650925    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy"
Nov 03 20:19:49.651244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.650934    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92a40ff-88c1-4ba0-99cf-47b494fe2ebe" containerName="istio-proxy"
Nov 03 20:19:49.656764 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.656738    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"]
Nov 03 20:19:49.656932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.656842    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.684660 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.683773    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-cad531\"/\"auth-test-model-cad531-kserve-self-signed-certs\""
Nov 03 20:19:49.684660 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.683903    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-cad531\"/\"default-dockercfg-ww4lg\""
Nov 03 20:19:49.684660 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.683968    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-cad531\"/\"kube-root-ca.crt\""
Nov 03 20:19:49.684660 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.684058    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-cad531\"/\"openshift-service-ca.crt\""
Nov 03 20:19:49.842596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.842276    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-home\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.842596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.842585    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tmp-dir\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.842875 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.842619    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-dshm\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.842875 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.842667    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tls-certs\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.842875 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.842719    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pkwv\" (UniqueName: \"kubernetes.io/projected/d0694eaa-d338-40b8-96e1-0fb4e54be0de-kube-api-access-4pkwv\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.842875 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.842816    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-model-cache\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.953512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.951741    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tmp-dir\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.953512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.951789    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-dshm\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.953512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.951844    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tls-certs\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.953512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.951897    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pkwv\" (UniqueName: \"kubernetes.io/projected/d0694eaa-d338-40b8-96e1-0fb4e54be0de-kube-api-access-4pkwv\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.953512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.952002    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-model-cache\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.953512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.952035    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-home\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.955200 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.955170    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tmp-dir\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.962305 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.962273    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-dshm\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.963172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.963147    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tls-certs\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.966129 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.964644    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-model-cache\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.966450 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.966410    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-home\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:49.967361 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:49.967324    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pkwv\" (UniqueName: \"kubernetes.io/projected/d0694eaa-d338-40b8-96e1-0fb4e54be0de-kube-api-access-4pkwv\") pod \"auth-test-model-cad531-kserve-84574d86d5-kcw9k\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") " pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:50.009517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:50.009474    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:50.304519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:50.304485    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"]
Nov 03 20:19:50.751671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:50.751635    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k" event={"ID":"d0694eaa-d338-40b8-96e1-0fb4e54be0de","Type":"ContainerStarted","Data":"66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1"}
Nov 03 20:19:50.751671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:50.751678    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k" event={"ID":"d0694eaa-d338-40b8-96e1-0fb4e54be0de","Type":"ContainerStarted","Data":"964f648ac1c3823504aa26afd49364ddccd0fbe9ccd53362b65cd883f15434c6"}
Nov 03 20:19:50.771891 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:50.771833    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k" podStartSLOduration=1.771819984 podStartE2EDuration="1.771819984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:50.769759107 +0000 UTC m=+2373.282580890" watchObservedRunningTime="2026-11-03 20:19:50.771819984 +0000 UTC m=+2373.284641763"
Nov 03 20:19:50.940458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:50.936311    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl" podUID="4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" containerName="main" containerID="cri-o://03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7" gracePeriod=45
Nov 03 20:19:51.009752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.009652    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:19:51.303077 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.303046    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:19:51.394194 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.394133    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-dshm\") pod \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") "
Nov 03 20:19:51.394366 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.394256    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-model-cache\") pod \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") "
Nov 03 20:19:51.394366 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.394341    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-kube-api-access-v5qlg\" (UniqueName: \"kubernetes.io/projected/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-kube-api-access-v5qlg\") pod \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") "
Nov 03 20:19:51.394557 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.394370    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tls-certs\") pod \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") "
Nov 03 20:19:51.394557 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.394497    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-home\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-home\") pod \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") "
Nov 03 20:19:51.394557 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.394537    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tmp-dir\") pod \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\" (UID: \"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5\") "
Nov 03 20:19:51.394967 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.394935    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-model-cache" pod "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" (UID: "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:51.395508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.395427    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-home" pod "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" (UID: "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:51.395508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.395485    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:51.396344 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.395685    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tmp-dir" pod "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" (UID: "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:51.401302 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.401272    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-dshm" pod "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" (UID: "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:51.401415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.401305    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-kube-api-access-v5qlg" pod "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" (UID: "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5"). InnerVolumeSpecName "kube-api-access-v5qlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:51.403310 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.403270    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tls-certs" pod "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" (UID: "4b68c73f-e9cc-4e95-83ac-5c95a778f6f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:51.498209 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.498157    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:51.498209 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.498203    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5qlg\" (UniqueName: \"kubernetes.io/projected/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-kube-api-access-v5qlg\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:51.498541 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.498219    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:51.498541 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.498234    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:51.498541 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.498246    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:51.607162 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.607062    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"]
Nov 03 20:19:51.850547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.850497    5600 generic.go:356] "Generic (PLEG): container finished" podID="4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" containerID="03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7" exitCode=0
Nov 03 20:19:51.852864 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.852825    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"
Nov 03 20:19:51.864140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.856986    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl" event={"ID":"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5","Type":"ContainerDied","Data":"03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7"}
Nov 03 20:19:51.864140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.862610    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl" event={"ID":"4b68c73f-e9cc-4e95-83ac-5c95a778f6f5","Type":"ContainerDied","Data":"043b22f559a4b4bc02ca0384a3d6d525d88a0222ae91cd32189ef815d57700c2"}
Nov 03 20:19:51.864140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.862641    5600 scope.go:122] "RemoveContainer" containerID="03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7"
Nov 03 20:19:51.892348 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.888800    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"]
Nov 03 20:19:51.892348 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.889317    5600 scope.go:122] "RemoveContainer" containerID="03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7"
Nov 03 20:19:51.905520 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:19:51.905278    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7\": container with ID starting with 03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7 not found: ID does not exist" containerID="03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7"
Nov 03 20:19:51.905520 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.905324    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7"} err="failed to get container status \"03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7\": rpc error: code = NotFound desc = could not find container \"03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7\": container with ID starting with 03ac2746c612d28536c8e19c36ad6abe6fdc392cf1eb904bda6fc6648071b6c7 not found: ID does not exist"
Nov 03 20:19:51.906177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:51.906148    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2092ba/test-model-2092ba-kserve-69c88c4895-dngbl"]
Nov 03 20:19:53.214837 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.214796    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"]
Nov 03 20:19:53.215887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.215861    5600 cpu_manager.go:414] "containerMap: removing container" podUID="4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" containerName="main"
Nov 03 20:19:53.215887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.215888    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" containerName="main"
Nov 03 20:19:53.216072 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.215950    5600 cpu_manager.go:414] "containerMap: removing container" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy"
Nov 03 20:19:53.216072 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.215958    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec08d5e-0c0e-459e-85b4-0b80b5b60b35" containerName="istio-proxy"
Nov 03 20:19:53.216181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.216107    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" containerName="main"
Nov 03 20:19:53.222823 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.222789    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.227973 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.227942    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-27d3ad-serving-cert\""
Nov 03 20:19:53.228819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.228776    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"]
Nov 03 20:19:53.378636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.378597    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-maas-api-tls\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.378827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.378807    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f9474d-1418-4a81-a37c-b0eb9b6e145b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.378870 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.378842    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5thb\" (UniqueName: \"kubernetes.io/projected/03f9474d-1418-4a81-a37c-b0eb9b6e145b-kube-api-access-q5thb\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.379031 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.379011    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-metrics-certs\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.480652 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.480564    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-maas-api-tls\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.480827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.480701    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f9474d-1418-4a81-a37c-b0eb9b6e145b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.480827 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:19:53.480726    5600 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-serving-cert: secret "maas-api-e2e-inf-b-27d3ad-serving-cert" not found
Nov 03 20:19:53.480827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.480735    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5thb\" (UniqueName: \"kubernetes.io/projected/03f9474d-1418-4a81-a37c-b0eb9b6e145b-kube-api-access-q5thb\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.480827 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:19:53.480794    5600 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-maas-api-tls podName:03f9474d-1418-4a81-a37c-b0eb9b6e145b nodeName:}" failed. No retries permitted until 2026-11-03 20:19:53.980772976 +0000 UTC m=+2376.493594756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-maas-api-tls") pod "maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9" (UID: "03f9474d-1418-4a81-a37c-b0eb9b6e145b") : secret "maas-api-e2e-inf-b-27d3ad-serving-cert" not found
Nov 03 20:19:53.481073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.480860    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-metrics-certs\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.487508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.481545    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f9474d-1418-4a81-a37c-b0eb9b6e145b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.487508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.485876    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-metrics-certs\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.491968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.491942    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5thb\" (UniqueName: \"kubernetes.io/projected/03f9474d-1418-4a81-a37c-b0eb9b6e145b-kube-api-access-q5thb\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:53.637671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.637624    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b68c73f-e9cc-4e95-83ac-5c95a778f6f5" path="/var/lib/kubelet/pods/4b68c73f-e9cc-4e95-83ac-5c95a778f6f5/volumes"
Nov 03 20:19:53.991460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:53.989597    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-maas-api-tls\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:54.021539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:54.013635    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-maas-api-tls\") pod \"maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:54.144368 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:54.143770    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:54.208549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:54.208457    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"]
Nov 03 20:19:54.208764 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:54.208732    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy" containerID="cri-o://3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0" gracePeriod=30
Nov 03 20:19:54.567466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:54.565733    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"]
Nov 03 20:19:54.606365 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:54.606032    5600 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:19:54.879618 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:54.879534    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9" event={"ID":"03f9474d-1418-4a81-a37c-b0eb9b6e145b","Type":"ContainerStarted","Data":"c6e9aa53ddf7b2e272079dee2dab9ee775ffedc65f84b312999b18f7a6bf5085"}
Nov 03 20:19:55.892872 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:55.892832    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9" event={"ID":"03f9474d-1418-4a81-a37c-b0eb9b6e145b","Type":"ContainerStarted","Data":"e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2"}
Nov 03 20:19:55.893376 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:55.892932    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:19:55.918634 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:55.918569    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9" podStartSLOduration=2.389275195 podStartE2EDuration="2.918546235s" totalImagesPullingTime="529.27104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:55.911533796 +0000 UTC m=+2378.424355578" watchObservedRunningTime="2026-11-03 20:19:55.918546235 +0000 UTC m=+2378.431368035"
Nov 03 20:19:58.800065 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:19:58.800024    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:19:59.611946 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.611917    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:19:59.696707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.696666    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-credential-socket\") pod \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") "
Nov 03 20:19:59.696921 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.696729    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istiod-ca-cert\") pod \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") "
Nov 03 20:19:59.696921 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.696863    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-certs\") pod \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") "
Nov 03 20:19:59.697057 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.696939    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-token\" (UniqueName: \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-token\") pod \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") "
Nov 03 20:19:59.697057 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.696952    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-credential-socket" pod "6d0b2608-2249-49b3-a2bc-57aa8d01d739" (UID: "6d0b2608-2249-49b3-a2bc-57aa8d01d739"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:59.697057 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.696988    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-socket\") pod \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") "
Nov 03 20:19:59.697222 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.697065    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-data\") pod \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") "
Nov 03 20:19:59.697222 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.697148    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-certs" pod "6d0b2608-2249-49b3-a2bc-57aa8d01d739" (UID: "6d0b2608-2249-49b3-a2bc-57aa8d01d739"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:59.697222 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.697210    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-envoy\") pod \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") "
Nov 03 20:19:59.697386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.697245    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-kube-api-access-wgckd\" (UniqueName: \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-kube-api-access-wgckd\") pod \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") "
Nov 03 20:19:59.697386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.697288    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-podinfo\") pod \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\" (UID: \"6d0b2608-2249-49b3-a2bc-57aa8d01d739\") "
Nov 03 20:19:59.697386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.697330    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-socket" pod "6d0b2608-2249-49b3-a2bc-57aa8d01d739" (UID: "6d0b2608-2249-49b3-a2bc-57aa8d01d739"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:59.698012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.697986    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.698121 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.698015    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.698121 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.698030    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.698254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.698217    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-data" pod "6d0b2608-2249-49b3-a2bc-57aa8d01d739" (UID: "6d0b2608-2249-49b3-a2bc-57aa8d01d739"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:59.698342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.698313    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istiod-ca-cert" pod "6d0b2608-2249-49b3-a2bc-57aa8d01d739" (UID: "6d0b2608-2249-49b3-a2bc-57aa8d01d739"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:59.700924 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.700888    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-kube-api-access-wgckd" pod "6d0b2608-2249-49b3-a2bc-57aa8d01d739" (UID: "6d0b2608-2249-49b3-a2bc-57aa8d01d739"). InnerVolumeSpecName "kube-api-access-wgckd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:59.704444 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.704403    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-envoy" pod "6d0b2608-2249-49b3-a2bc-57aa8d01d739" (UID: "6d0b2608-2249-49b3-a2bc-57aa8d01d739"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:59.704753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.704732    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-podinfo" pod "6d0b2608-2249-49b3-a2bc-57aa8d01d739" (UID: "6d0b2608-2249-49b3-a2bc-57aa8d01d739"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:59.704839 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.704770    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-token" pod "6d0b2608-2249-49b3-a2bc-57aa8d01d739" (UID: "6d0b2608-2249-49b3-a2bc-57aa8d01d739"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:59.799629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.799590    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.799629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.799629    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.799896 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.799646    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgckd\" (UniqueName: \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-kube-api-access-wgckd\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.799896 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.799663    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.799896 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.799679    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.799896 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.799693    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0b2608-2249-49b3-a2bc-57aa8d01d739-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.940238 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.940202    5600 generic.go:356] "Generic (PLEG): container finished" podID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerID="3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0" exitCode=0
Nov 03 20:19:59.940734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.940266    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"
Nov 03 20:19:59.940734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.940292    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" event={"ID":"6d0b2608-2249-49b3-a2bc-57aa8d01d739","Type":"ContainerDied","Data":"3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0"}
Nov 03 20:19:59.940734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.940356    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6" event={"ID":"6d0b2608-2249-49b3-a2bc-57aa8d01d739","Type":"ContainerDied","Data":"82cc4a633354ea7e145e8314b056bd8c5cafd936a62e8a680e90fd0503113651"}
Nov 03 20:19:59.940734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.940382    5600 scope.go:122] "RemoveContainer" containerID="3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0"
Nov 03 20:19:59.952944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.952924    5600 scope.go:122] "RemoveContainer" containerID="3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0"
Nov 03 20:19:59.953288 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:19:59.953263    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0\": container with ID starting with 3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0 not found: ID does not exist" containerID="3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0"
Nov 03 20:19:59.953356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.953298    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0"} err="failed to get container status \"3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0\": rpc error: code = NotFound desc = could not find container \"3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0\": container with ID starting with 3d0fdb040724c571394024d3bb546fe417098a74b8941f4cb5bc3cac589b69e0 not found: ID does not exist"
Nov 03 20:19:59.965717 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.965683    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"]
Nov 03 20:19:59.969683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:19:59.969653    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-6hxv6"]
Nov 03 20:20:00.014245 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:00.012053    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:20:00.036678 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:00.036650    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:20:00.541327 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:00.539938    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"]
Nov 03 20:20:00.541327 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:00.540631    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d" podUID="ebac9276-2fb6-4924-9b00-5e5e589ba683" containerName="maas-api" containerID="cri-o://4033461df0e38a887361737786f5cd6a6450aff29491222d5bde00a1a4629e2a" gracePeriod=30
Nov 03 20:20:01.001573 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:00.990862    5600 generic.go:356] "Generic (PLEG): container finished" podID="ebac9276-2fb6-4924-9b00-5e5e589ba683" containerID="4033461df0e38a887361737786f5cd6a6450aff29491222d5bde00a1a4629e2a" exitCode=0
Nov 03 20:20:01.001573 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:00.993306    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d" event={"ID":"ebac9276-2fb6-4924-9b00-5e5e589ba683","Type":"ContainerDied","Data":"4033461df0e38a887361737786f5cd6a6450aff29491222d5bde00a1a4629e2a"}
Nov 03 20:20:01.258422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.258332    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:20:01.493820 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.493793    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:20:01.558800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.558692    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebac9276-2fb6-4924-9b00-5e5e589ba683-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebac9276-2fb6-4924-9b00-5e5e589ba683-openshift-service-ca\") pod \"ebac9276-2fb6-4924-9b00-5e5e589ba683\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") "
Nov 03 20:20:01.559116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.559085    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-maas-api-tls\") pod \"ebac9276-2fb6-4924-9b00-5e5e589ba683\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") "
Nov 03 20:20:01.559225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.559145    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebac9276-2fb6-4924-9b00-5e5e589ba683-openshift-service-ca" pod "ebac9276-2fb6-4924-9b00-5e5e589ba683" (UID: "ebac9276-2fb6-4924-9b00-5e5e589ba683"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:01.559225 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.559174    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-metrics-certs\") pod \"ebac9276-2fb6-4924-9b00-5e5e589ba683\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") "
Nov 03 20:20:01.559350 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.559272    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebac9276-2fb6-4924-9b00-5e5e589ba683-kube-api-access-csnll\" (UniqueName: \"kubernetes.io/projected/ebac9276-2fb6-4924-9b00-5e5e589ba683-kube-api-access-csnll\") pod \"ebac9276-2fb6-4924-9b00-5e5e589ba683\" (UID: \"ebac9276-2fb6-4924-9b00-5e5e589ba683\") "
Nov 03 20:20:01.559877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.559855    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebac9276-2fb6-4924-9b00-5e5e589ba683-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:01.561856 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.561824    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-maas-api-tls" pod "ebac9276-2fb6-4924-9b00-5e5e589ba683" (UID: "ebac9276-2fb6-4924-9b00-5e5e589ba683"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:01.562321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.562302    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-metrics-certs" pod "ebac9276-2fb6-4924-9b00-5e5e589ba683" (UID: "ebac9276-2fb6-4924-9b00-5e5e589ba683"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:01.562591 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.562568    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebac9276-2fb6-4924-9b00-5e5e589ba683-kube-api-access-csnll" pod "ebac9276-2fb6-4924-9b00-5e5e589ba683" (UID: "ebac9276-2fb6-4924-9b00-5e5e589ba683"). InnerVolumeSpecName "kube-api-access-csnll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:01.624547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.623400    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" path="/var/lib/kubelet/pods/6d0b2608-2249-49b3-a2bc-57aa8d01d739/volumes"
Nov 03 20:20:01.661815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.661770    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:01.661815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.661814    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csnll\" (UniqueName: \"kubernetes.io/projected/ebac9276-2fb6-4924-9b00-5e5e589ba683-kube-api-access-csnll\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:01.661815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.661826    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebac9276-2fb6-4924-9b00-5e5e589ba683-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:01.919644 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:01.919607    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:20:02.019733 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:02.018656    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"
Nov 03 20:20:02.019733 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:02.019240    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d" event={"ID":"ebac9276-2fb6-4924-9b00-5e5e589ba683","Type":"ContainerDied","Data":"a94fdb2ec555239c2aa4b202c951793d04d4a58ec6af0de59c7857a6cc1eb242"}
Nov 03 20:20:02.019733 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:02.019292    5600 scope.go:122] "RemoveContainer" containerID="4033461df0e38a887361737786f5cd6a6450aff29491222d5bde00a1a4629e2a"
Nov 03 20:20:02.050172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:02.050134    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"]
Nov 03 20:20:02.055376 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:02.055348    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2092ba-557c6dcb8-jmc6d"]
Nov 03 20:20:02.099723 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:02.099685    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"]
Nov 03 20:20:02.726458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:02.713329    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"]
Nov 03 20:20:02.726458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:02.713634    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" podUID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" containerName="istio-proxy" containerID="cri-o://7cb94c8dcd5c53ad2c018d96ee6671adc11d73c3a8d217e1ed2fc4b6934b925b" gracePeriod=30
Nov 03 20:20:02.736458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:02.730413    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"]
Nov 03 20:20:02.736458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:02.730713    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" podUID="10e2387a-d83a-4df4-a7c9-781356645f2b" containerName="istio-proxy" containerID="cri-o://40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398" gracePeriod=30
Nov 03 20:20:03.614997 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:03.614933    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf" podUID="849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" containerName="main" containerID="cri-o://f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6" gracePeriod=45
Nov 03 20:20:03.627102 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:03.627066    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebac9276-2fb6-4924-9b00-5e5e589ba683" path="/var/lib/kubelet/pods/ebac9276-2fb6-4924-9b00-5e5e589ba683/volumes"
Nov 03 20:20:03.844617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:03.844409    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf" podUID="849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.121:8000/ready\": dial tcp 10.134.0.121:8000: connect: connection refused"
Nov 03 20:20:04.019836 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.019730    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:20:04.046400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.046366    5600 generic.go:356] "Generic (PLEG): container finished" podID="849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" containerID="f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6" exitCode=0
Nov 03 20:20:04.046597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.046422    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"
Nov 03 20:20:04.046597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.046466    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf" event={"ID":"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb","Type":"ContainerDied","Data":"f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6"}
Nov 03 20:20:04.046597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.046511    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf" event={"ID":"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb","Type":"ContainerDied","Data":"d453bce951a5d3b154922a963c6de612facbe91fbbeedbf1c03c18aafad91a62"}
Nov 03 20:20:04.046597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.046532    5600 scope.go:122] "RemoveContainer" containerID="f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6"
Nov 03 20:20:04.052253 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.052225    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-dshm\") pod \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") "
Nov 03 20:20:04.052418 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.052274    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-model-cache\") pod \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") "
Nov 03 20:20:04.052503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.052467    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tls-certs\") pod \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") "
Nov 03 20:20:04.052675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.052634    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-home\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-home\") pod \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") "
Nov 03 20:20:04.052880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.052688    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-kube-api-access-cn2bl\" (UniqueName: \"kubernetes.io/projected/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-kube-api-access-cn2bl\") pod \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") "
Nov 03 20:20:04.052880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.052763    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tmp-dir\") pod \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\" (UID: \"849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb\") "
Nov 03 20:20:04.052880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.052830    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-model-cache" pod "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" (UID: "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:04.053248 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.053174    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tmp-dir" pod "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" (UID: "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:04.053681 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.053657    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:04.053836 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.053685    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:04.053836 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.053469    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-home" pod "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" (UID: "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:04.062678 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.062478    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tls-certs" pod "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" (UID: "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:04.063113 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.063042    5600 scope.go:122] "RemoveContainer" containerID="f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6"
Nov 03 20:20:04.063599 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:20:04.063574    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6\": container with ID starting with f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6 not found: ID does not exist" containerID="f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6"
Nov 03 20:20:04.063792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.063762    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6"} err="failed to get container status \"f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6\": rpc error: code = NotFound desc = could not find container \"f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6\": container with ID starting with f0ddf2c35838cbc760f40e44aabe6c6dab15732317cd3c1ffe36b113aae86ef6 not found: ID does not exist"
Nov 03 20:20:04.065968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.065940    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-kube-api-access-cn2bl" pod "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" (UID: "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb"). InnerVolumeSpecName "kube-api-access-cn2bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:04.066201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.066159    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-dshm" pod "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" (UID: "849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:04.155947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.155907    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:04.155947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.155941    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn2bl\" (UniqueName: \"kubernetes.io/projected/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-kube-api-access-cn2bl\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:04.155947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.155954    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:04.156214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.155965    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:04.469455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.469398    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"]
Nov 03 20:20:04.477495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:04.477412    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9b1aed/test-model-9b1aed-kserve-796bdc449f-w5ddf"]
Nov 03 20:20:05.620935 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:05.620891    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" path="/var/lib/kubelet/pods/849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb/volumes"
Nov 03 20:20:06.301647 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.301600    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"]
Nov 03 20:20:06.303307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.303266    5600 cpu_manager.go:414] "containerMap: removing container" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy"
Nov 03 20:20:06.303534 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.303515    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy"
Nov 03 20:20:06.303655 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.303643    5600 cpu_manager.go:414] "containerMap: removing container" podUID="849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" containerName="main"
Nov 03 20:20:06.303753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.303740    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" containerName="main"
Nov 03 20:20:06.303874 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.303863    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ebac9276-2fb6-4924-9b00-5e5e589ba683" containerName="maas-api"
Nov 03 20:20:06.303958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.303947    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebac9276-2fb6-4924-9b00-5e5e589ba683" containerName="maas-api"
Nov 03 20:20:06.304299 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.304285    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0b2608-2249-49b3-a2bc-57aa8d01d739" containerName="istio-proxy"
Nov 03 20:20:06.304393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.304383    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="849ff6ad-84b0-4846-8d4a-a31b0b3ee2eb" containerName="main"
Nov 03 20:20:06.304504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.304494    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebac9276-2fb6-4924-9b00-5e5e589ba683" containerName="maas-api"
Nov 03 20:20:06.311377 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.311347    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"]
Nov 03 20:20:06.311733 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.311705    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.321697 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.321636    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-ee2323\"/\"default-dockercfg-xkxg2\""
Nov 03 20:20:06.325239 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.324738    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-ee2323\"/\"auth-test-model-ee2323-kserve-self-signed-certs\""
Nov 03 20:20:06.375024 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.374987    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-ee2323\"/\"kube-root-ca.crt\""
Nov 03 20:20:06.375024 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.374997    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-ee2323\"/\"openshift-service-ca.crt\""
Nov 03 20:20:06.436491 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.436449    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0deab001-abc3-432e-982f-8b3d0f16008b-tls-certs\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.436708 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.436563    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-tmp-dir\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.436708 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.436654    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-dshm\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.436840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.436771    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbgl\" (UniqueName: \"kubernetes.io/projected/0deab001-abc3-432e-982f-8b3d0f16008b-kube-api-access-hpbgl\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.437015 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.436987    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-home\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.437123 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.437105    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-model-cache\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.544971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.542849    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-model-cache\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.544971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.543119    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0deab001-abc3-432e-982f-8b3d0f16008b-tls-certs\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.544971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.543225    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-tmp-dir\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.544971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.543291    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-dshm\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.544971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.543363    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbgl\" (UniqueName: \"kubernetes.io/projected/0deab001-abc3-432e-982f-8b3d0f16008b-kube-api-access-hpbgl\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.544971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.543521    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-home\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.544971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.544066    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-home\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.544971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.544373    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-model-cache\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.546454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.546211    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-tmp-dir\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.581517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.577149    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0deab001-abc3-432e-982f-8b3d0f16008b-tls-certs\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.581517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.578018    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-dshm\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.605555 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.605518    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbgl\" (UniqueName: \"kubernetes.io/projected/0deab001-abc3-432e-982f-8b3d0f16008b-kube-api-access-hpbgl\") pod \"auth-test-model-ee2323-kserve-86547c9c7c-t5rbr\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") " pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.652899 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.652713    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:06.667524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.667426    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd" podUID="605b8ced-a357-40a6-8d96-4c051b215128" containerName="main" containerID="cri-o://a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d" gracePeriod=45
Nov 03 20:20:06.681961 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.681623    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:20:06.681961 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.681684    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" podUID="10e2387a-d83a-4df4-a7c9-781356645f2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:20:06.974504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.974472    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"]
Nov 03 20:20:06.978081 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:20:06.977698    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0deab001_abc3_432e_982f_8b3d0f16008b.slice/crio-7ee04ac1d20765d3102bb5df3c9779eafac31176038d8888b9376d0f5d1640b8 WatchSource:0}: Error finding container 7ee04ac1d20765d3102bb5df3c9779eafac31176038d8888b9376d0f5d1640b8: Status 404 returned error can't find the container with id 7ee04ac1d20765d3102bb5df3c9779eafac31176038d8888b9376d0f5d1640b8
Nov 03 20:20:06.996400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:06.996369    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:20:07.090004 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.089001    5600 generic.go:356] "Generic (PLEG): container finished" podID="605b8ced-a357-40a6-8d96-4c051b215128" containerID="a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d" exitCode=0
Nov 03 20:20:07.090004 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.089203    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd" event={"ID":"605b8ced-a357-40a6-8d96-4c051b215128","Type":"ContainerDied","Data":"a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d"}
Nov 03 20:20:07.090004 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.089237    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd" event={"ID":"605b8ced-a357-40a6-8d96-4c051b215128","Type":"ContainerDied","Data":"35affd17a82960d045949e24cfe2fedbe880105cd4b2751bceb58b2d9139fedd"}
Nov 03 20:20:07.090004 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.089258    5600 scope.go:122] "RemoveContainer" containerID="a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d"
Nov 03 20:20:07.090004 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.089399    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"
Nov 03 20:20:07.100565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.094859    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr" event={"ID":"0deab001-abc3-432e-982f-8b3d0f16008b","Type":"ContainerStarted","Data":"7ee04ac1d20765d3102bb5df3c9779eafac31176038d8888b9376d0f5d1640b8"}
Nov 03 20:20:07.100565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.096615    5600 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:20:07.100565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.096675    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" podUID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:20:07.100565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.100393    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n" podUID="d0ac6eff-118d-4cd7-9a34-f173552e6bc0" containerName="payload-processing" containerID="cri-o://8bb30f334055c1d9522c6c907e83ae626c261c6213fb46cf7f1f8136df27ccfa" gracePeriod=25
Nov 03 20:20:07.114610 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.114573    5600 scope.go:122] "RemoveContainer" containerID="a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d"
Nov 03 20:20:07.115965 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:20:07.115917    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d\": container with ID starting with a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d not found: ID does not exist" containerID="a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d"
Nov 03 20:20:07.116043 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.115977    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d"} err="failed to get container status \"a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d\": rpc error: code = NotFound desc = could not find container \"a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d\": container with ID starting with a2bb2669ed803644c74b34cc99ebcd4ec2a1f6a2e6a3658e57ce13267e9b6c4d not found: ID does not exist"
Nov 03 20:20:07.158583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.158543    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-model-cache\") pod \"605b8ced-a357-40a6-8d96-4c051b215128\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") "
Nov 03 20:20:07.159118 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.159050    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-model-cache" pod "605b8ced-a357-40a6-8d96-4c051b215128" (UID: "605b8ced-a357-40a6-8d96-4c051b215128"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:07.159357 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.159328    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/605b8ced-a357-40a6-8d96-4c051b215128-tls-certs\" (UniqueName: \"kubernetes.io/secret/605b8ced-a357-40a6-8d96-4c051b215128-tls-certs\") pod \"605b8ced-a357-40a6-8d96-4c051b215128\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") "
Nov 03 20:20:07.159427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.159410    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-dshm\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-dshm\") pod \"605b8ced-a357-40a6-8d96-4c051b215128\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") "
Nov 03 20:20:07.159550 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.159535    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-tmp-dir\") pod \"605b8ced-a357-40a6-8d96-4c051b215128\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") "
Nov 03 20:20:07.159666 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.159574    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-home\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-home\") pod \"605b8ced-a357-40a6-8d96-4c051b215128\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") "
Nov 03 20:20:07.159666 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.159640    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605b8ced-a357-40a6-8d96-4c051b215128-kube-api-access-5mc4s\" (UniqueName: \"kubernetes.io/projected/605b8ced-a357-40a6-8d96-4c051b215128-kube-api-access-5mc4s\") pod \"605b8ced-a357-40a6-8d96-4c051b215128\" (UID: \"605b8ced-a357-40a6-8d96-4c051b215128\") "
Nov 03 20:20:07.160267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.160243    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:07.161125 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.161089    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-home" pod "605b8ced-a357-40a6-8d96-4c051b215128" (UID: "605b8ced-a357-40a6-8d96-4c051b215128"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:07.161578 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.161557    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-tmp-dir" pod "605b8ced-a357-40a6-8d96-4c051b215128" (UID: "605b8ced-a357-40a6-8d96-4c051b215128"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:07.163657 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.163533    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605b8ced-a357-40a6-8d96-4c051b215128-tls-certs" pod "605b8ced-a357-40a6-8d96-4c051b215128" (UID: "605b8ced-a357-40a6-8d96-4c051b215128"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:07.163657 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.163587    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-dshm" pod "605b8ced-a357-40a6-8d96-4c051b215128" (UID: "605b8ced-a357-40a6-8d96-4c051b215128"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:07.163657 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.163637    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605b8ced-a357-40a6-8d96-4c051b215128-kube-api-access-5mc4s" pod "605b8ced-a357-40a6-8d96-4c051b215128" (UID: "605b8ced-a357-40a6-8d96-4c051b215128"). InnerVolumeSpecName "kube-api-access-5mc4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:07.262264 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.262223    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:07.262264 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.262259    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:07.262264 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.262269    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mc4s\" (UniqueName: \"kubernetes.io/projected/605b8ced-a357-40a6-8d96-4c051b215128-kube-api-access-5mc4s\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:07.262591 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.262278    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605b8ced-a357-40a6-8d96-4c051b215128-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:07.262591 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.262287    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605b8ced-a357-40a6-8d96-4c051b215128-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:07.417732 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.417638    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"]
Nov 03 20:20:07.421170 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.421140    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-babea4/test-model-babea4-kserve-84fb77dc7-cnqzd"]
Nov 03 20:20:07.621273 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:07.621231    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605b8ced-a357-40a6-8d96-4c051b215128" path="/var/lib/kubelet/pods/605b8ced-a357-40a6-8d96-4c051b215128/volumes"
Nov 03 20:20:08.084813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.084720    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:20:08.122291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.120266    5600 generic.go:356] "Generic (PLEG): container finished" podID="10e2387a-d83a-4df4-a7c9-781356645f2b" containerID="40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398" exitCode=0
Nov 03 20:20:08.122291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.120479    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" event={"ID":"10e2387a-d83a-4df4-a7c9-781356645f2b","Type":"ContainerDied","Data":"40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398"}
Nov 03 20:20:08.122291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.120531    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk" event={"ID":"10e2387a-d83a-4df4-a7c9-781356645f2b","Type":"ContainerDied","Data":"1d8aec9930bde9915c3101aad6fff35db69f9c38d258f7153e10b1ff964f203a"}
Nov 03 20:20:08.122291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.120561    5600 scope.go:122] "RemoveContainer" containerID="40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398"
Nov 03 20:20:08.122291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.120678    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"
Nov 03 20:20:08.129632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.126121    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr" event={"ID":"0deab001-abc3-432e-982f-8b3d0f16008b","Type":"ContainerStarted","Data":"571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1"}
Nov 03 20:20:08.129719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.129634    5600 generic.go:356] "Generic (PLEG): container finished" podID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" containerID="7cb94c8dcd5c53ad2c018d96ee6671adc11d73c3a8d217e1ed2fc4b6934b925b" exitCode=0
Nov 03 20:20:08.130102 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.129782    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" event={"ID":"5bc0b3b1-a009-4a94-a0e7-77b7683548e7","Type":"ContainerDied","Data":"7cb94c8dcd5c53ad2c018d96ee6671adc11d73c3a8d217e1ed2fc4b6934b925b"}
Nov 03 20:20:08.179472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.179312    5600 scope.go:122] "RemoveContainer" containerID="40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398"
Nov 03 20:20:08.182774 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:20:08.182736    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398\": container with ID starting with 40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398 not found: ID does not exist" containerID="40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398"
Nov 03 20:20:08.182929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.182782    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398"} err="failed to get container status \"40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398\": rpc error: code = NotFound desc = could not find container \"40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398\": container with ID starting with 40238b6da60256a174600494f1a0e7895dc24f18328d0e1e13e87760993e2398 not found: ID does not exist"
Nov 03 20:20:08.183272 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.183246    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-socket\") pod \"10e2387a-d83a-4df4-a7c9-781356645f2b\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") "
Nov 03 20:20:08.183363 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.183292    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-credential-socket\") pod \"10e2387a-d83a-4df4-a7c9-781356645f2b\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") "
Nov 03 20:20:08.183363 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.183336    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-data\") pod \"10e2387a-d83a-4df4-a7c9-781356645f2b\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") "
Nov 03 20:20:08.183498 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.183387    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-kube-api-access-89tsj\" (UniqueName: \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-kube-api-access-89tsj\") pod \"10e2387a-d83a-4df4-a7c9-781356645f2b\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") "
Nov 03 20:20:08.186764 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.186735    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-credential-socket" pod "10e2387a-d83a-4df4-a7c9-781356645f2b" (UID: "10e2387a-d83a-4df4-a7c9-781356645f2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.187025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.186992    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-socket" pod "10e2387a-d83a-4df4-a7c9-781356645f2b" (UID: "10e2387a-d83a-4df4-a7c9-781356645f2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.187408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.187188    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-data" pod "10e2387a-d83a-4df4-a7c9-781356645f2b" (UID: "10e2387a-d83a-4df4-a7c9-781356645f2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.199537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.198169    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-token\" (UniqueName: \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-token\") pod \"10e2387a-d83a-4df4-a7c9-781356645f2b\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") "
Nov 03 20:20:08.199767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.199750    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e2387a-d83a-4df4-a7c9-781356645f2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e2387a-d83a-4df4-a7c9-781356645f2b-istiod-ca-cert\") pod \"10e2387a-d83a-4df4-a7c9-781356645f2b\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") "
Nov 03 20:20:08.200374 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.200261    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e2387a-d83a-4df4-a7c9-781356645f2b-istiod-ca-cert" pod "10e2387a-d83a-4df4-a7c9-781356645f2b" (UID: "10e2387a-d83a-4df4-a7c9-781356645f2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:08.200624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.200609    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-envoy\") pod \"10e2387a-d83a-4df4-a7c9-781356645f2b\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") "
Nov 03 20:20:08.200805 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.200643    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-kube-api-access-89tsj" pod "10e2387a-d83a-4df4-a7c9-781356645f2b" (UID: "10e2387a-d83a-4df4-a7c9-781356645f2b"). InnerVolumeSpecName "kube-api-access-89tsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:08.200953 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.200938    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-certs\") pod \"10e2387a-d83a-4df4-a7c9-781356645f2b\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") "
Nov 03 20:20:08.201472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.201365    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-certs" pod "10e2387a-d83a-4df4-a7c9-781356645f2b" (UID: "10e2387a-d83a-4df4-a7c9-781356645f2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.201620 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.201606    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-podinfo\") pod \"10e2387a-d83a-4df4-a7c9-781356645f2b\" (UID: \"10e2387a-d83a-4df4-a7c9-781356645f2b\") "
Nov 03 20:20:08.201971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.201943    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-token" pod "10e2387a-d83a-4df4-a7c9-781356645f2b" (UID: "10e2387a-d83a-4df4-a7c9-781356645f2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:08.202829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.202808    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.203072 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.203057    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.203168 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.203157    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.203249 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.203239    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89tsj\" (UniqueName: \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-kube-api-access-89tsj\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.203329 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.203319    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.203400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.203390    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e2387a-d83a-4df4-a7c9-781356645f2b-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.203508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.203495    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.203586 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.203389    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-envoy" pod "10e2387a-d83a-4df4-a7c9-781356645f2b" (UID: "10e2387a-d83a-4df4-a7c9-781356645f2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.205148 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.205117    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-podinfo" pod "10e2387a-d83a-4df4-a7c9-781356645f2b" (UID: "10e2387a-d83a-4df4-a7c9-781356645f2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:08.228368 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.228340    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:20:08.255901 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.255832    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr" podStartSLOduration=2.255815319 podStartE2EDuration="2.255815319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:08.182516294 +0000 UTC m=+2390.695338091" watchObservedRunningTime="2026-11-03 20:20:08.255815319 +0000 UTC m=+2390.768637102"
Nov 03 20:20:08.325328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.306376    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.327542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.327513    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e2387a-d83a-4df4-a7c9-781356645f2b-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.428595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.428555    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-socket\") pod \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") "
Nov 03 20:20:08.428796 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.428620    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-kube-api-access-d2gfp\" (UniqueName: \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-kube-api-access-d2gfp\") pod \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") "
Nov 03 20:20:08.428796 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.428672    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-data\") pod \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") "
Nov 03 20:20:08.428796 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.428723    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-credential-socket\") pod \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") "
Nov 03 20:20:08.428796 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.428787    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-certs\") pod \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") "
Nov 03 20:20:08.429026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.428864    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-socket" pod "5bc0b3b1-a009-4a94-a0e7-77b7683548e7" (UID: "5bc0b3b1-a009-4a94-a0e7-77b7683548e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.429026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.428889    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istiod-ca-cert\") pod \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") "
Nov 03 20:20:08.429026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.428983    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-token\" (UniqueName: \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-token\") pod \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") "
Nov 03 20:20:08.429189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.429067    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-envoy\") pod \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") "
Nov 03 20:20:08.429189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.429074    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-data" pod "5bc0b3b1-a009-4a94-a0e7-77b7683548e7" (UID: "5bc0b3b1-a009-4a94-a0e7-77b7683548e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.429663 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.429638    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-certs" pod "5bc0b3b1-a009-4a94-a0e7-77b7683548e7" (UID: "5bc0b3b1-a009-4a94-a0e7-77b7683548e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.429765 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.429743    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-podinfo\") pod \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\" (UID: \"5bc0b3b1-a009-4a94-a0e7-77b7683548e7\") "
Nov 03 20:20:08.430540 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.430460    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.430540 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.430484    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.430540 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.430498    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.430883 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.430852    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-credential-socket" pod "5bc0b3b1-a009-4a94-a0e7-77b7683548e7" (UID: "5bc0b3b1-a009-4a94-a0e7-77b7683548e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.431104 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.431078    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istiod-ca-cert" pod "5bc0b3b1-a009-4a94-a0e7-77b7683548e7" (UID: "5bc0b3b1-a009-4a94-a0e7-77b7683548e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:08.432928 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.432880    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-podinfo" pod "5bc0b3b1-a009-4a94-a0e7-77b7683548e7" (UID: "5bc0b3b1-a009-4a94-a0e7-77b7683548e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:08.433021 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.432929    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-token" pod "5bc0b3b1-a009-4a94-a0e7-77b7683548e7" (UID: "5bc0b3b1-a009-4a94-a0e7-77b7683548e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:08.434739 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.434720    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-kube-api-access-d2gfp" pod "5bc0b3b1-a009-4a94-a0e7-77b7683548e7" (UID: "5bc0b3b1-a009-4a94-a0e7-77b7683548e7"). InnerVolumeSpecName "kube-api-access-d2gfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:08.436618 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.436597    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-envoy" pod "5bc0b3b1-a009-4a94-a0e7-77b7683548e7" (UID: "5bc0b3b1-a009-4a94-a0e7-77b7683548e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.453853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.453802    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"]
Nov 03 20:20:08.458608 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.458571    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-tt2tk"]
Nov 03 20:20:08.532577 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.532534    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2gfp\" (UniqueName: \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-kube-api-access-d2gfp\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.532577 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.532580    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.532852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.532597    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.532852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.532610    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.532852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.532627    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.532852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.532640    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc0b3b1-a009-4a94-a0e7-77b7683548e7-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.653276 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:08.653237    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:09.143264 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:09.143237    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"
Nov 03 20:20:09.143840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:09.143276    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5" event={"ID":"5bc0b3b1-a009-4a94-a0e7-77b7683548e7","Type":"ContainerDied","Data":"003e246b735d68c4002f2fa5a6ac78bebba650a300962610e9d9f06aea671dd4"}
Nov 03 20:20:09.143840 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:09.143342    5600 scope.go:122] "RemoveContainer" containerID="7cb94c8dcd5c53ad2c018d96ee6671adc11d73c3a8d217e1ed2fc4b6934b925b"
Nov 03 20:20:09.182459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:09.174040    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"]
Nov 03 20:20:09.195933 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:09.195896    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-lv8f5"]
Nov 03 20:20:09.648200 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:09.648162    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e2387a-d83a-4df4-a7c9-781356645f2b" path="/var/lib/kubelet/pods/10e2387a-d83a-4df4-a7c9-781356645f2b/volumes"
Nov 03 20:20:09.652405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:09.652310    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" path="/var/lib/kubelet/pods/5bc0b3b1-a009-4a94-a0e7-77b7683548e7/volumes"
Nov 03 20:20:11.176540 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:11.176500    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"]
Nov 03 20:20:11.177089 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:11.176789    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs" podUID="34d8f77d-4a5f-479e-8c60-32d52a23b253" containerName="maas-api" containerID="cri-o://c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9" gracePeriod=30
Nov 03 20:20:11.911606 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:11.911579    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:20:12.050820 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.050753    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d8f77d-4a5f-479e-8c60-32d52a23b253-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d8f77d-4a5f-479e-8c60-32d52a23b253-openshift-service-ca\") pod \"34d8f77d-4a5f-479e-8c60-32d52a23b253\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") "
Nov 03 20:20:12.051057 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.050886    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d8f77d-4a5f-479e-8c60-32d52a23b253-kube-api-access-zfppt\" (UniqueName: \"kubernetes.io/projected/34d8f77d-4a5f-479e-8c60-32d52a23b253-kube-api-access-zfppt\") pod \"34d8f77d-4a5f-479e-8c60-32d52a23b253\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") "
Nov 03 20:20:12.051057 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.050935    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-maas-api-tls\") pod \"34d8f77d-4a5f-479e-8c60-32d52a23b253\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") "
Nov 03 20:20:12.051057 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.051028    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-metrics-certs\") pod \"34d8f77d-4a5f-479e-8c60-32d52a23b253\" (UID: \"34d8f77d-4a5f-479e-8c60-32d52a23b253\") "
Nov 03 20:20:12.051876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.051842    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d8f77d-4a5f-479e-8c60-32d52a23b253-openshift-service-ca" pod "34d8f77d-4a5f-479e-8c60-32d52a23b253" (UID: "34d8f77d-4a5f-479e-8c60-32d52a23b253"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:12.054009 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.053977    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d8f77d-4a5f-479e-8c60-32d52a23b253-kube-api-access-zfppt" pod "34d8f77d-4a5f-479e-8c60-32d52a23b253" (UID: "34d8f77d-4a5f-479e-8c60-32d52a23b253"). InnerVolumeSpecName "kube-api-access-zfppt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:12.054624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.054597    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-maas-api-tls" pod "34d8f77d-4a5f-479e-8c60-32d52a23b253" (UID: "34d8f77d-4a5f-479e-8c60-32d52a23b253"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:12.055177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.055156    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-metrics-certs" pod "34d8f77d-4a5f-479e-8c60-32d52a23b253" (UID: "34d8f77d-4a5f-479e-8c60-32d52a23b253"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:12.153009 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.152975    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d8f77d-4a5f-479e-8c60-32d52a23b253-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:12.153009 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.153007    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfppt\" (UniqueName: \"kubernetes.io/projected/34d8f77d-4a5f-479e-8c60-32d52a23b253-kube-api-access-zfppt\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:12.153009 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.153018    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:12.153252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.153027    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d8f77d-4a5f-479e-8c60-32d52a23b253-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:12.179482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.179448    5600 generic.go:356] "Generic (PLEG): container finished" podID="34d8f77d-4a5f-479e-8c60-32d52a23b253" containerID="c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9" exitCode=0
Nov 03 20:20:12.179937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.179518    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"
Nov 03 20:20:12.179937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.179531    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs" event={"ID":"34d8f77d-4a5f-479e-8c60-32d52a23b253","Type":"ContainerDied","Data":"c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9"}
Nov 03 20:20:12.179937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.179592    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs" event={"ID":"34d8f77d-4a5f-479e-8c60-32d52a23b253","Type":"ContainerDied","Data":"c7895b17a8a8d91266ea4dab4a1b753366ade28787510cac70bd8daf06c02d5a"}
Nov 03 20:20:12.179937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.179634    5600 scope.go:122] "RemoveContainer" containerID="c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9"
Nov 03 20:20:12.193617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.193595    5600 scope.go:122] "RemoveContainer" containerID="c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9"
Nov 03 20:20:12.194084 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:20:12.194054    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9\": container with ID starting with c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9 not found: ID does not exist" containerID="c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9"
Nov 03 20:20:12.194192 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.194096    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9"} err="failed to get container status \"c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9\": rpc error: code = NotFound desc = could not find container \"c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9\": container with ID starting with c8b16585bcbf19aaec49cdb5fc74ba399abed858d54400b920712a5ece4e34f9 not found: ID does not exist"
Nov 03 20:20:12.224405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.224371    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"]
Nov 03 20:20:12.228351 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:12.228323    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-663013-64ff7dd99-tjjhs"]
Nov 03 20:20:13.621401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:13.621363    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d8f77d-4a5f-479e-8c60-32d52a23b253" path="/var/lib/kubelet/pods/34d8f77d-4a5f-479e-8c60-32d52a23b253/volumes"
Nov 03 20:20:15.792258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:15.792220    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n" podUID="d0ac6eff-118d-4cd7-9a34-f173552e6bc0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:16.653684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:16.653649    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:16.670948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:16.670918    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:17.265208 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:17.265177    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:20:18.875861 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:18.875826    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"]
Nov 03 20:20:18.876415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:18.876155    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy" containerID="cri-o://9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827" gracePeriod=30
Nov 03 20:20:21.533190 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.533150    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"]
Nov 03 20:20:21.534074 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534049    5600 cpu_manager.go:414] "containerMap: removing container" podUID="605b8ced-a357-40a6-8d96-4c051b215128" containerName="main"
Nov 03 20:20:21.534074 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534075    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605b8ced-a357-40a6-8d96-4c051b215128" containerName="main"
Nov 03 20:20:21.534259 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534104    5600 cpu_manager.go:414] "containerMap: removing container" podUID="10e2387a-d83a-4df4-a7c9-781356645f2b" containerName="istio-proxy"
Nov 03 20:20:21.534259 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534113    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e2387a-d83a-4df4-a7c9-781356645f2b" containerName="istio-proxy"
Nov 03 20:20:21.534259 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534126    5600 cpu_manager.go:414] "containerMap: removing container" podUID="34d8f77d-4a5f-479e-8c60-32d52a23b253" containerName="maas-api"
Nov 03 20:20:21.534259 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534134    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d8f77d-4a5f-479e-8c60-32d52a23b253" containerName="maas-api"
Nov 03 20:20:21.534259 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534152    5600 cpu_manager.go:414] "containerMap: removing container" podUID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" containerName="istio-proxy"
Nov 03 20:20:21.534259 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534159    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" containerName="istio-proxy"
Nov 03 20:20:21.534613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534332    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="605b8ced-a357-40a6-8d96-4c051b215128" containerName="main"
Nov 03 20:20:21.534613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534352    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e2387a-d83a-4df4-a7c9-781356645f2b" containerName="istio-proxy"
Nov 03 20:20:21.534613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534361    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d8f77d-4a5f-479e-8c60-32d52a23b253" containerName="maas-api"
Nov 03 20:20:21.534613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.534373    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc0b3b1-a009-4a94-a0e7-77b7683548e7" containerName="istio-proxy"
Nov 03 20:20:21.540922 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.540893    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.548874 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.548841    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"]
Nov 03 20:20:21.553517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.553493    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-a476f8\"/\"openshift-service-ca.crt\""
Nov 03 20:20:21.553678 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.553566    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-a476f8\"/\"default-dockercfg-z9cmr\""
Nov 03 20:20:21.553767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.553697    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-a476f8\"/\"auth-test-model-a476f8-kserve-self-signed-certs\""
Nov 03 20:20:21.553829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.553767    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-a476f8\"/\"kube-root-ca.crt\""
Nov 03 20:20:21.673042 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.673006    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-tmp-dir\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.673240 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.673065    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-home\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.673240 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.673144    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8ck\" (UniqueName: \"kubernetes.io/projected/2f6e46a8-7993-4245-9e36-92669fe70f34-kube-api-access-cs8ck\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.673332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.673262    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6e46a8-7993-4245-9e36-92669fe70f34-tls-certs\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.673390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.673376    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-dshm\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.673501 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.673483    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-model-cache\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.777255 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.775463    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-dshm\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.777255 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.775604    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-model-cache\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.777255 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.775764    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-tmp-dir\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.777255 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.775836    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-home\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.777255 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.775878    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8ck\" (UniqueName: \"kubernetes.io/projected/2f6e46a8-7993-4245-9e36-92669fe70f34-kube-api-access-cs8ck\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.777255 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.775949    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6e46a8-7993-4245-9e36-92669fe70f34-tls-certs\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.777255 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.776413    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-home\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.777255 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.776693    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-model-cache\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.777255 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.777101    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-tmp-dir\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.790660 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.787972    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8ck\" (UniqueName: \"kubernetes.io/projected/2f6e46a8-7993-4245-9e36-92669fe70f34-kube-api-access-cs8ck\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.805459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.797122    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6e46a8-7993-4245-9e36-92669fe70f34-tls-certs\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.805459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.797390    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-dshm\") pod \"auth-test-model-a476f8-kserve-74bf477bb9-vx2w9\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") " pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:21.868063 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:21.867776    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:22.222911 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:22.222821    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"]
Nov 03 20:20:22.296449 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:22.296391    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9" event={"ID":"2f6e46a8-7993-4245-9e36-92669fe70f34","Type":"ContainerStarted","Data":"3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338"}
Nov 03 20:20:22.296635 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:22.296456    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9" event={"ID":"2f6e46a8-7993-4245-9e36-92669fe70f34","Type":"ContainerStarted","Data":"3dc76d6f02f8ba9af5e9514ebfed9668fe29d7d6e8850cdf427c3ac14d6aeec8"}
Nov 03 20:20:22.313489 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:22.313411    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9" podStartSLOduration=1.313395135 podStartE2EDuration="1.313395135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:22.312078741 +0000 UTC m=+2404.824900524" watchObservedRunningTime="2026-11-03 20:20:22.313395135 +0000 UTC m=+2404.826216917"
Nov 03 20:20:22.869269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:22.869233    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:24.236398 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.236371    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:20:24.315963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.315935    5600 generic.go:356] "Generic (PLEG): container finished" podID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerID="9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827" exitCode=0
Nov 03 20:20:24.320106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.320063    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"
Nov 03 20:20:24.321345 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.321317    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" event={"ID":"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec","Type":"ContainerDied","Data":"9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827"}
Nov 03 20:20:24.321608 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.321575    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz" event={"ID":"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec","Type":"ContainerDied","Data":"5d70213a35e82614b250bfdb1af3e6d4f320f900265c66ee538696d96bc4d798"}
Nov 03 20:20:24.321817 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.321759    5600 scope.go:122] "RemoveContainer" containerID="9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827"
Nov 03 20:20:24.337297 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.337263    5600 scope.go:122] "RemoveContainer" containerID="cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830"
Nov 03 20:20:24.340422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.340394    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-socket\") pod \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") "
Nov 03 20:20:24.340595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.340561    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-certs\") pod \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") "
Nov 03 20:20:24.340670 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.340636    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-credential-socket\") pod \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") "
Nov 03 20:20:24.340732 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.340705    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-token\" (UniqueName: \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-token\") pod \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") "
Nov 03 20:20:24.340798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.340786    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-podinfo\") pod \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") "
Nov 03 20:20:24.340854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.340815    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-envoy\") pod \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") "
Nov 03 20:20:24.340910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.340862    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-data\") pod \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") "
Nov 03 20:20:24.340985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.340966    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istiod-ca-cert\") pod \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") "
Nov 03 20:20:24.341051 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.341003    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-kube-api-access-ftflt\" (UniqueName: \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-kube-api-access-ftflt\") pod \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\" (UID: \"7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec\") "
Nov 03 20:20:24.341551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.341468    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-certs" pod "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" (UID: "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.341551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.341488    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istiod-ca-cert" pod "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" (UID: "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:24.342149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.341733    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.342149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.341759    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.342149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.341925    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-socket" pod "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" (UID: "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.342149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.342120    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-credential-socket" pod "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" (UID: "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.342498 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.342393    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-data" pod "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" (UID: "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.350696 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.349114    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-kube-api-access-ftflt" pod "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" (UID: "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec"). InnerVolumeSpecName "kube-api-access-ftflt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:24.351675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.351643    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-envoy" pod "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" (UID: "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.351831 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.351682    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-podinfo" pod "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" (UID: "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:24.351954 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.351845    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-token" pod "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" (UID: "7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:24.357426 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.357296    5600 scope.go:122] "RemoveContainer" containerID="9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827"
Nov 03 20:20:24.357923 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:20:24.357890    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827\": container with ID starting with 9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827 not found: ID does not exist" containerID="9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827"
Nov 03 20:20:24.358017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.357937    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827"} err="failed to get container status \"9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827\": rpc error: code = NotFound desc = could not find container \"9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827\": container with ID starting with 9daf5217179214d7532378b0602c195ac989e3c4f95320243618984d00984827 not found: ID does not exist"
Nov 03 20:20:24.358017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.357962    5600 scope.go:122] "RemoveContainer" containerID="cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830"
Nov 03 20:20:24.358351 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:20:24.358318    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830\": container with ID starting with cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830 not found: ID does not exist" containerID="cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830"
Nov 03 20:20:24.358572 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.358508    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830"} err="failed to get container status \"cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830\": rpc error: code = NotFound desc = could not find container \"cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830\": container with ID starting with cc8d2a75c20a11242c73585632fb570ec8fe30abd568b5dfde36fb592155a830 not found: ID does not exist"
Nov 03 20:20:24.443380 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.443331    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.443380 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.443373    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.443380 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.443387    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.443667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.443402    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.443667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.443416    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.443667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.443446    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftflt\" (UniqueName: \"kubernetes.io/projected/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-kube-api-access-ftflt\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.443667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.443461    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.671590 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.671550    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"]
Nov 03 20:20:24.675778 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:24.675737    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-663013-openshift-default-8678b5b749-2rsrz"]
Nov 03 20:20:25.650731 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:25.650692    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" path="/var/lib/kubelet/pods/7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec/volumes"
Nov 03 20:20:25.793340 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:25.793294    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n" podUID="d0ac6eff-118d-4cd7-9a34-f173552e6bc0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:27.387156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:27.385786    5600 generic.go:356] "Generic (PLEG): container finished" podID="d0ac6eff-118d-4cd7-9a34-f173552e6bc0" containerID="8bb30f334055c1d9522c6c907e83ae626c261c6213fb46cf7f1f8136df27ccfa" exitCode=0
Nov 03 20:20:27.387156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:27.385858    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n" event={"ID":"d0ac6eff-118d-4cd7-9a34-f173552e6bc0","Type":"ContainerDied","Data":"8bb30f334055c1d9522c6c907e83ae626c261c6213fb46cf7f1f8136df27ccfa"}
Nov 03 20:20:27.467212 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:27.467186    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:20:27.612427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:27.612380    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-plugins-config-volume\") pod \"d0ac6eff-118d-4cd7-9a34-f173552e6bc0\" (UID: \"d0ac6eff-118d-4cd7-9a34-f173552e6bc0\") "
Nov 03 20:20:27.612715 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:27.612521    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-kube-api-access-8b5nd\" (UniqueName: \"kubernetes.io/projected/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-kube-api-access-8b5nd\") pod \"d0ac6eff-118d-4cd7-9a34-f173552e6bc0\" (UID: \"d0ac6eff-118d-4cd7-9a34-f173552e6bc0\") "
Nov 03 20:20:27.613058 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:27.612964    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-plugins-config-volume" pod "d0ac6eff-118d-4cd7-9a34-f173552e6bc0" (UID: "d0ac6eff-118d-4cd7-9a34-f173552e6bc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:27.613275 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:27.613236    5600 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-plugins-config-volume\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:27.616223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:27.616191    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-kube-api-access-8b5nd" pod "d0ac6eff-118d-4cd7-9a34-f173552e6bc0" (UID: "d0ac6eff-118d-4cd7-9a34-f173552e6bc0"). InnerVolumeSpecName "kube-api-access-8b5nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:27.714598 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:27.714495    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b5nd\" (UniqueName: \"kubernetes.io/projected/d0ac6eff-118d-4cd7-9a34-f173552e6bc0-kube-api-access-8b5nd\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:28.395697 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:28.395666    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"
Nov 03 20:20:28.396220 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:28.395708    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n" event={"ID":"d0ac6eff-118d-4cd7-9a34-f173552e6bc0","Type":"ContainerDied","Data":"e27b2831f992594ffad259751c28d41c1f9ed48c556067e305d7841153133c55"}
Nov 03 20:20:28.396220 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:28.395774    5600 scope.go:122] "RemoveContainer" containerID="8bb30f334055c1d9522c6c907e83ae626c261c6213fb46cf7f1f8136df27ccfa"
Nov 03 20:20:28.416734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:28.416692    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"]
Nov 03 20:20:28.420784 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:28.420746    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-663013-848b6d9486-s696n"]
Nov 03 20:20:28.800267 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:20:28.800173    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:20:29.619304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:29.619269    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ac6eff-118d-4cd7-9a34-f173552e6bc0" path="/var/lib/kubelet/pods/d0ac6eff-118d-4cd7-9a34-f173552e6bc0/volumes"
Nov 03 20:20:31.882478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:31.882316    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:31.901189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:31.901149    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"]
Nov 03 20:20:31.963386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:31.963353    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:32.450463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.450401    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7" podUID="c93ba2d7-918c-45c3-b5ca-67803135937c" containerName="maas-api" containerID="cri-o://da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea" gracePeriod=30
Nov 03 20:20:32.478805 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.478759    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:20:32.831560 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.831526    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:20:32.938334 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.938284    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93ba2d7-918c-45c3-b5ca-67803135937c-kube-api-access-fxvvd\" (UniqueName: \"kubernetes.io/projected/c93ba2d7-918c-45c3-b5ca-67803135937c-kube-api-access-fxvvd\") pod \"c93ba2d7-918c-45c3-b5ca-67803135937c\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") "
Nov 03 20:20:32.938897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.938379    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-maas-api-tls\") pod \"c93ba2d7-918c-45c3-b5ca-67803135937c\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") "
Nov 03 20:20:32.938897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.938462    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-metrics-certs\") pod \"c93ba2d7-918c-45c3-b5ca-67803135937c\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") "
Nov 03 20:20:32.938897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.938603    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c93ba2d7-918c-45c3-b5ca-67803135937c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93ba2d7-918c-45c3-b5ca-67803135937c-openshift-service-ca\") pod \"c93ba2d7-918c-45c3-b5ca-67803135937c\" (UID: \"c93ba2d7-918c-45c3-b5ca-67803135937c\") "
Nov 03 20:20:32.939071 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.938994    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93ba2d7-918c-45c3-b5ca-67803135937c-openshift-service-ca" pod "c93ba2d7-918c-45c3-b5ca-67803135937c" (UID: "c93ba2d7-918c-45c3-b5ca-67803135937c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:32.939815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.939785    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93ba2d7-918c-45c3-b5ca-67803135937c-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:32.941153 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.941128    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93ba2d7-918c-45c3-b5ca-67803135937c-kube-api-access-fxvvd" pod "c93ba2d7-918c-45c3-b5ca-67803135937c" (UID: "c93ba2d7-918c-45c3-b5ca-67803135937c"). InnerVolumeSpecName "kube-api-access-fxvvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:32.941366 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.941348    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-metrics-certs" pod "c93ba2d7-918c-45c3-b5ca-67803135937c" (UID: "c93ba2d7-918c-45c3-b5ca-67803135937c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:32.946296 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:32.946263    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-maas-api-tls" pod "c93ba2d7-918c-45c3-b5ca-67803135937c" (UID: "c93ba2d7-918c-45c3-b5ca-67803135937c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:33.041904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.041859    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxvvd\" (UniqueName: \"kubernetes.io/projected/c93ba2d7-918c-45c3-b5ca-67803135937c-kube-api-access-fxvvd\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:33.041904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.041895    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:33.041904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.041910    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c93ba2d7-918c-45c3-b5ca-67803135937c-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:33.460058 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.460016    5600 generic.go:356] "Generic (PLEG): container finished" podID="c93ba2d7-918c-45c3-b5ca-67803135937c" containerID="da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea" exitCode=0
Nov 03 20:20:33.460258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.460073    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"
Nov 03 20:20:33.460399 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.460374    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7" event={"ID":"c93ba2d7-918c-45c3-b5ca-67803135937c","Type":"ContainerDied","Data":"da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea"}
Nov 03 20:20:33.460595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.460578    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7" event={"ID":"c93ba2d7-918c-45c3-b5ca-67803135937c","Type":"ContainerDied","Data":"94441590a3c72e1a7f77de728f4229c88088d19b05d0ab75eec060c07cf62b3c"}
Nov 03 20:20:33.460706 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.460694    5600 scope.go:122] "RemoveContainer" containerID="da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea"
Nov 03 20:20:33.474680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.474464    5600 scope.go:122] "RemoveContainer" containerID="da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea"
Nov 03 20:20:33.474944 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:20:33.474915    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea\": container with ID starting with da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea not found: ID does not exist" containerID="da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea"
Nov 03 20:20:33.475022 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.474958    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea"} err="failed to get container status \"da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea\": rpc error: code = NotFound desc = could not find container \"da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea\": container with ID starting with da2873cbd75ea05f764db0ae4ce7a39307799528de8f5d4dd43a87b89e7371ea not found: ID does not exist"
Nov 03 20:20:33.496548 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.496502    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"]
Nov 03 20:20:33.502291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.502256    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-5348ca-6bbc7dbfb9-jhbx7"]
Nov 03 20:20:33.636209 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:33.635949    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c93ba2d7-918c-45c3-b5ca-67803135937c" path="/var/lib/kubelet/pods/c93ba2d7-918c-45c3-b5ca-67803135937c/volumes"
Nov 03 20:20:35.242464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.242224    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"]
Nov 03 20:20:35.242913 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.242591    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x" podUID="d21b7271-831b-42e4-8494-68277b1d0e10" containerName="maas-api" containerID="cri-o://9682ebfbb505f6d4ddec5639a726d99c06db90af8d5b7ff2f7e6f43c51041b6c" gracePeriod=30
Nov 03 20:20:35.513854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.513759    5600 generic.go:356] "Generic (PLEG): container finished" podID="d21b7271-831b-42e4-8494-68277b1d0e10" containerID="9682ebfbb505f6d4ddec5639a726d99c06db90af8d5b7ff2f7e6f43c51041b6c" exitCode=0
Nov 03 20:20:35.514036 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.514006    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x" event={"ID":"d21b7271-831b-42e4-8494-68277b1d0e10","Type":"ContainerDied","Data":"9682ebfbb505f6d4ddec5639a726d99c06db90af8d5b7ff2f7e6f43c51041b6c"}
Nov 03 20:20:35.909224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.909194    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:20:35.941206 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.941136    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"]
Nov 03 20:20:35.942627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.942578    5600 cpu_manager.go:414] "containerMap: removing container" podUID="d0ac6eff-118d-4cd7-9a34-f173552e6bc0" containerName="payload-processing"
Nov 03 20:20:35.942838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.942819    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ac6eff-118d-4cd7-9a34-f173552e6bc0" containerName="payload-processing"
Nov 03 20:20:35.942968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.942944    5600 cpu_manager.go:414] "containerMap: removing container" podUID="c93ba2d7-918c-45c3-b5ca-67803135937c" containerName="maas-api"
Nov 03 20:20:35.943075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.943061    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c93ba2d7-918c-45c3-b5ca-67803135937c" containerName="maas-api"
Nov 03 20:20:35.943258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.943244    5600 cpu_manager.go:414] "containerMap: removing container" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy"
Nov 03 20:20:35.943380 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.943366    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy"
Nov 03 20:20:35.943534 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.943518    5600 cpu_manager.go:414] "containerMap: removing container" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy"
Nov 03 20:20:35.943667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.943653    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy"
Nov 03 20:20:35.943760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.943749    5600 cpu_manager.go:414] "containerMap: removing container" podUID="d21b7271-831b-42e4-8494-68277b1d0e10" containerName="maas-api"
Nov 03 20:20:35.943838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.943829    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21b7271-831b-42e4-8494-68277b1d0e10" containerName="maas-api"
Nov 03 20:20:35.944084 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.944068    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="c93ba2d7-918c-45c3-b5ca-67803135937c" containerName="maas-api"
Nov 03 20:20:35.944206 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.944195    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy"
Nov 03 20:20:35.944291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.944281    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21b7271-831b-42e4-8494-68277b1d0e10" containerName="maas-api"
Nov 03 20:20:35.944392 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.944381    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ac6eff-118d-4cd7-9a34-f173552e6bc0" containerName="payload-processing"
Nov 03 20:20:35.951184 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.951155    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"]
Nov 03 20:20:35.959535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.958104    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:35.992544 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.991874    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-27d3ad\"/\"default-dockercfg-4lf8q\""
Nov 03 20:20:35.992544 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.991979    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-27d3ad\"/\"kube-root-ca.crt\""
Nov 03 20:20:35.992544 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.992023    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-27d3ad\"/\"openshift-service-ca.crt\""
Nov 03 20:20:35.992544 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:35.992101    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-27d3ad\"/\"test-model-27d3ad-kserve-self-signed-certs\""
Nov 03 20:20:36.043949 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.043909    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21b7271-831b-42e4-8494-68277b1d0e10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21b7271-831b-42e4-8494-68277b1d0e10-openshift-service-ca\") pod \"d21b7271-831b-42e4-8494-68277b1d0e10\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") "
Nov 03 20:20:36.044162 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.043984    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-maas-api-tls\") pod \"d21b7271-831b-42e4-8494-68277b1d0e10\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") "
Nov 03 20:20:36.044162 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.044018    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-metrics-certs\") pod \"d21b7271-831b-42e4-8494-68277b1d0e10\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") "
Nov 03 20:20:36.044296 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.044171    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21b7271-831b-42e4-8494-68277b1d0e10-kube-api-access-686h6\" (UniqueName: \"kubernetes.io/projected/d21b7271-831b-42e4-8494-68277b1d0e10-kube-api-access-686h6\") pod \"d21b7271-831b-42e4-8494-68277b1d0e10\" (UID: \"d21b7271-831b-42e4-8494-68277b1d0e10\") "
Nov 03 20:20:36.044469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.044421    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-home\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.045621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.045595    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjg77\" (UniqueName: \"kubernetes.io/projected/3c3c6285-c426-4c53-affa-ea450c508260-kube-api-access-sjg77\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.045916 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.045890    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-tmp-dir\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.046108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.046087    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-dshm\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.046252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.046235    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-model-cache\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.046584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.046409    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3c6285-c426-4c53-affa-ea450c508260-tls-certs\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.046886 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.044989    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21b7271-831b-42e4-8494-68277b1d0e10-openshift-service-ca" pod "d21b7271-831b-42e4-8494-68277b1d0e10" (UID: "d21b7271-831b-42e4-8494-68277b1d0e10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:36.047823 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.047792    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21b7271-831b-42e4-8494-68277b1d0e10-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:36.052736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.052704    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-maas-api-tls" pod "d21b7271-831b-42e4-8494-68277b1d0e10" (UID: "d21b7271-831b-42e4-8494-68277b1d0e10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:36.052884 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.052716    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-metrics-certs" pod "d21b7271-831b-42e4-8494-68277b1d0e10" (UID: "d21b7271-831b-42e4-8494-68277b1d0e10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:36.052884 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.052849    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21b7271-831b-42e4-8494-68277b1d0e10-kube-api-access-686h6" pod "d21b7271-831b-42e4-8494-68277b1d0e10" (UID: "d21b7271-831b-42e4-8494-68277b1d0e10"). InnerVolumeSpecName "kube-api-access-686h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:36.169517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.169276    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-tmp-dir\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.169517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.169351    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-dshm\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.169517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.169375    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-model-cache\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.169517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.169469    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3c6285-c426-4c53-affa-ea450c508260-tls-certs\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.171480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.170242    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-home\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.171480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.170363    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjg77\" (UniqueName: \"kubernetes.io/projected/3c3c6285-c426-4c53-affa-ea450c508260-kube-api-access-sjg77\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.171480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.170517    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:36.171480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.170536    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21b7271-831b-42e4-8494-68277b1d0e10-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:36.171480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.170554    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-686h6\" (UniqueName: \"kubernetes.io/projected/d21b7271-831b-42e4-8494-68277b1d0e10-kube-api-access-686h6\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:36.171480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.171164    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-home\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.172454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.172203    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-model-cache\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.172684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.172634    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-tmp-dir\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.183459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.173026    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3c6285-c426-4c53-affa-ea450c508260-tls-certs\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.183459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.175812    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-dshm\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.201638 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.196142    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjg77\" (UniqueName: \"kubernetes.io/projected/3c3c6285-c426-4c53-affa-ea450c508260-kube-api-access-sjg77\") pod \"test-model-27d3ad-kserve-596675cdfc-tzf6l\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") " pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.322407 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.322373    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:36.560460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.560073    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x" event={"ID":"d21b7271-831b-42e4-8494-68277b1d0e10","Type":"ContainerDied","Data":"be7d6c76c3f0a5aa798ca17bcbec0d1da45d30b5c48c8798d45701c6d8bd6ad1"}
Nov 03 20:20:36.560460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.560156    5600 scope.go:122] "RemoveContainer" containerID="9682ebfbb505f6d4ddec5639a726d99c06db90af8d5b7ff2f7e6f43c51041b6c"
Nov 03 20:20:36.560460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.560315    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"
Nov 03 20:20:36.668357 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.668322    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"]
Nov 03 20:20:36.685328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.685256    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-babea4-7b57547fcd-gkb9x"]
Nov 03 20:20:36.736038 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:36.736013    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"]
Nov 03 20:20:36.739920 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:20:36.739844    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c3c6285_c426_4c53_affa_ea450c508260.slice/crio-b4627b9fe1cbdafdbb41d36a3adc50841d833e1ed7cc3f7dca485fc792fda01d WatchSource:0}: Error finding container b4627b9fe1cbdafdbb41d36a3adc50841d833e1ed7cc3f7dca485fc792fda01d: Status 404 returned error can't find the container with id b4627b9fe1cbdafdbb41d36a3adc50841d833e1ed7cc3f7dca485fc792fda01d
Nov 03 20:20:37.593258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:37.593153    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l" event={"ID":"3c3c6285-c426-4c53-affa-ea450c508260","Type":"ContainerStarted","Data":"481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026"}
Nov 03 20:20:37.593258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:37.593230    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l" event={"ID":"3c3c6285-c426-4c53-affa-ea450c508260","Type":"ContainerStarted","Data":"b4627b9fe1cbdafdbb41d36a3adc50841d833e1ed7cc3f7dca485fc792fda01d"}
Nov 03 20:20:37.619146 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:37.619095    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l" podStartSLOduration=2.619079938 podStartE2EDuration="2.619079938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:37.61439627 +0000 UTC m=+2420.127218053" watchObservedRunningTime="2026-11-03 20:20:37.619079938 +0000 UTC m=+2420.131901721"
Nov 03 20:20:37.622039 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:37.622003    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21b7271-831b-42e4-8494-68277b1d0e10" path="/var/lib/kubelet/pods/d21b7271-831b-42e4-8494-68277b1d0e10/volumes"
Nov 03 20:20:38.325518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:38.323543    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:40.440800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:40.440755    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"]
Nov 03 20:20:40.441278 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:40.441012    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy" containerID="cri-o://5157ed0588b6bb223360454f3c1c24d8500d866bd4f0a2193dc0fff35fe66f95" gracePeriod=30
Nov 03 20:20:43.018997 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.016394    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"]
Nov 03 20:20:43.019951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.019926    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdbf60d-a054-4c28-afa8-c83aa8e9f5ec" containerName="istio-proxy"
Nov 03 20:20:43.028803 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.028777    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.039086 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.039059    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-0ffd89\"/\"auth-test-model-0ffd89-kserve-self-signed-certs\""
Nov 03 20:20:43.040423 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.040299    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-0ffd89\"/\"default-dockercfg-4ssjn\""
Nov 03 20:20:43.042752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.042675    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-0ffd89\"/\"kube-root-ca.crt\""
Nov 03 20:20:43.043509 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.043473    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-0ffd89\"/\"openshift-service-ca.crt\""
Nov 03 20:20:43.044688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.044667    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"]
Nov 03 20:20:43.165515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.165427    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-model-cache\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.166275 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.166248    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-tmp-dir\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.166886 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.166864    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06da157a-f7c8-41e6-870a-a630afb5d912-tls-certs\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.167600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.167109    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-home\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.167600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.167295    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92d4b\" (UniqueName: \"kubernetes.io/projected/06da157a-f7c8-41e6-870a-a630afb5d912-kube-api-access-92d4b\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.167600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.167364    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-dshm\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.268967 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.268932    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-tmp-dir\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.269401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.269331    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06da157a-f7c8-41e6-870a-a630afb5d912-tls-certs\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.269401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.269385    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-home\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.269627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.269425    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92d4b\" (UniqueName: \"kubernetes.io/projected/06da157a-f7c8-41e6-870a-a630afb5d912-kube-api-access-92d4b\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.269627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.269530    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-dshm\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.269627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.269606    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-model-cache\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.270047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.270024    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-model-cache\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.270362 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.270343    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-tmp-dir\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.276284 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.275653    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-home\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.303420 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.303355    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-dshm\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.304239 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.304168    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92d4b\" (UniqueName: \"kubernetes.io/projected/06da157a-f7c8-41e6-870a-a630afb5d912-kube-api-access-92d4b\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.318217 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.316898    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06da157a-f7c8-41e6-870a-a630afb5d912-tls-certs\") pod \"auth-test-model-0ffd89-kserve-759768c8b5-87q9z\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") " pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.379608 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.378365    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:43.747298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:43.747266    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"]
Nov 03 20:20:43.749827 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:20:43.749791    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06da157a_f7c8_41e6_870a_a630afb5d912.slice/crio-a396b37ad2ba387a102a2bb6b69ad7829d2e7264968e9d9e0f9b8ed5427694c8 WatchSource:0}: Error finding container a396b37ad2ba387a102a2bb6b69ad7829d2e7264968e9d9e0f9b8ed5427694c8: Status 404 returned error can't find the container with id a396b37ad2ba387a102a2bb6b69ad7829d2e7264968e9d9e0f9b8ed5427694c8
Nov 03 20:20:44.759382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:44.759339    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z" event={"ID":"06da157a-f7c8-41e6-870a-a630afb5d912","Type":"ContainerStarted","Data":"9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e"}
Nov 03 20:20:44.759838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:44.759393    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z" event={"ID":"06da157a-f7c8-41e6-870a-a630afb5d912","Type":"ContainerStarted","Data":"a396b37ad2ba387a102a2bb6b69ad7829d2e7264968e9d9e0f9b8ed5427694c8"}
Nov 03 20:20:44.777262 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:44.777170    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z" podStartSLOduration=2.777150344 podStartE2EDuration="2.777150344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:44.775613174 +0000 UTC m=+2427.288434973" watchObservedRunningTime="2026-11-03 20:20:44.777150344 +0000 UTC m=+2427.289972127"
Nov 03 20:20:45.031184 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.031092    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"]
Nov 03 20:20:45.036516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.035936    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a865d6c2-f2tvc"]
Nov 03 20:20:45.380589 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.380506    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:45.625270 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.625233    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd38fad9-7ea8-4f12-8b53-747ed99e901b" path="/var/lib/kubelet/pods/fd38fad9-7ea8-4f12-8b53-747ed99e901b/volumes"
Nov 03 20:20:45.780622 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.778792    5600 generic.go:356] "Generic (PLEG): container finished" podID="e0d51280-043d-411f-aea0-4af2eb678c66" containerID="5157ed0588b6bb223360454f3c1c24d8500d866bd4f0a2193dc0fff35fe66f95" exitCode=0
Nov 03 20:20:45.780622 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.779672    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" event={"ID":"e0d51280-043d-411f-aea0-4af2eb678c66","Type":"ContainerDied","Data":"5157ed0588b6bb223360454f3c1c24d8500d866bd4f0a2193dc0fff35fe66f95"}
Nov 03 20:20:45.780622 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.779760    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm" event={"ID":"e0d51280-043d-411f-aea0-4af2eb678c66","Type":"ContainerDied","Data":"de0af7d92dba29cf3987ed2e8bd88ac26e337f0afa25baea70ad0e7279090a5f"}
Nov 03 20:20:45.780622 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.779806    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de0af7d92dba29cf3987ed2e8bd88ac26e337f0afa25baea70ad0e7279090a5f"
Nov 03 20:20:45.787197 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.787143    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:20:45.861504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.861393    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-istio-token\" (UniqueName: \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-istio-token\") pod \"e0d51280-043d-411f-aea0-4af2eb678c66\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") "
Nov 03 20:20:45.861504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.861459    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0d51280-043d-411f-aea0-4af2eb678c66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d51280-043d-411f-aea0-4af2eb678c66-istio-podinfo\") pod \"e0d51280-043d-411f-aea0-4af2eb678c66\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") "
Nov 03 20:20:45.861799 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.861523    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-socket\") pod \"e0d51280-043d-411f-aea0-4af2eb678c66\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") "
Nov 03 20:20:45.861799 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.861614    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-certs\") pod \"e0d51280-043d-411f-aea0-4af2eb678c66\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") "
Nov 03 20:20:45.865624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.863572    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-data\") pod \"e0d51280-043d-411f-aea0-4af2eb678c66\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") "
Nov 03 20:20:45.865624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.863656    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-kube-api-access-h4vlp\" (UniqueName: \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-kube-api-access-h4vlp\") pod \"e0d51280-043d-411f-aea0-4af2eb678c66\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") "
Nov 03 20:20:45.865624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.863687    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d51280-043d-411f-aea0-4af2eb678c66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d51280-043d-411f-aea0-4af2eb678c66-istiod-ca-cert\") pod \"e0d51280-043d-411f-aea0-4af2eb678c66\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") "
Nov 03 20:20:45.865624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.863770    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-envoy\") pod \"e0d51280-043d-411f-aea0-4af2eb678c66\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") "
Nov 03 20:20:45.865624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.863828    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-credential-socket\") pod \"e0d51280-043d-411f-aea0-4af2eb678c66\" (UID: \"e0d51280-043d-411f-aea0-4af2eb678c66\") "
Nov 03 20:20:45.865624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.864795    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-socket" pod "e0d51280-043d-411f-aea0-4af2eb678c66" (UID: "e0d51280-043d-411f-aea0-4af2eb678c66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:45.865624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.865060    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-data" pod "e0d51280-043d-411f-aea0-4af2eb678c66" (UID: "e0d51280-043d-411f-aea0-4af2eb678c66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:45.865624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.865289    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-credential-socket" pod "e0d51280-043d-411f-aea0-4af2eb678c66" (UID: "e0d51280-043d-411f-aea0-4af2eb678c66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:45.865624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.865345    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-certs" pod "e0d51280-043d-411f-aea0-4af2eb678c66" (UID: "e0d51280-043d-411f-aea0-4af2eb678c66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:45.866043 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.865697    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d51280-043d-411f-aea0-4af2eb678c66-istiod-ca-cert" pod "e0d51280-043d-411f-aea0-4af2eb678c66" (UID: "e0d51280-043d-411f-aea0-4af2eb678c66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:45.875606 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.875565    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-envoy" pod "e0d51280-043d-411f-aea0-4af2eb678c66" (UID: "e0d51280-043d-411f-aea0-4af2eb678c66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:45.875798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.875602    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0d51280-043d-411f-aea0-4af2eb678c66-istio-podinfo" pod "e0d51280-043d-411f-aea0-4af2eb678c66" (UID: "e0d51280-043d-411f-aea0-4af2eb678c66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:45.882680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.882627    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-kube-api-access-h4vlp" pod "e0d51280-043d-411f-aea0-4af2eb678c66" (UID: "e0d51280-043d-411f-aea0-4af2eb678c66"). InnerVolumeSpecName "kube-api-access-h4vlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:45.883661 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.883635    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-istio-token" pod "e0d51280-043d-411f-aea0-4af2eb678c66" (UID: "e0d51280-043d-411f-aea0-4af2eb678c66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:45.965694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.965655    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.965694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.965693    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4vlp\" (UniqueName: \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-kube-api-access-h4vlp\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.965976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.965710    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d51280-043d-411f-aea0-4af2eb678c66-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.965976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.965724    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.965976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.965738    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.965976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.965753    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d51280-043d-411f-aea0-4af2eb678c66-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.965976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.965766    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d51280-043d-411f-aea0-4af2eb678c66-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.965976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.965780    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.965976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:45.965793    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d51280-043d-411f-aea0-4af2eb678c66-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:46.326983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:46.326886    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:46.358527 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:46.358492    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:46.790305 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:46.790274    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"
Nov 03 20:20:46.815055 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:46.815023    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:20:46.829682 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:46.829652    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"]
Nov 03 20:20:46.834611 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:46.834582    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4rm"]
Nov 03 20:20:47.619983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:47.619948    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" path="/var/lib/kubelet/pods/e0d51280-043d-411f-aea0-4af2eb678c66/volumes"
Nov 03 20:20:50.420141 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.420098    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"]
Nov 03 20:20:50.421138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.421118    5600 cpu_manager.go:414] "containerMap: removing container" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy"
Nov 03 20:20:50.421208 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.421142    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy"
Nov 03 20:20:50.421359 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.421346    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d51280-043d-411f-aea0-4af2eb678c66" containerName="istio-proxy"
Nov 03 20:20:50.435920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.435888    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"]
Nov 03 20:20:50.436092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.436015    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.445616 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.445582    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-9ff9f5-serving-cert\""
Nov 03 20:20:50.476910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.476857    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.477098 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.476982    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8985a09b-1f37-4792-9588-389ecdafe353-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.477155 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.477140    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.477213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.477188    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqmx\" (UniqueName: \"kubernetes.io/projected/8985a09b-1f37-4792-9588-389ecdafe353-kube-api-access-fhqmx\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.578530 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.578500    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.578737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.578621    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8985a09b-1f37-4792-9588-389ecdafe353-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.578737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.578701    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.578861 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.578746    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqmx\" (UniqueName: \"kubernetes.io/projected/8985a09b-1f37-4792-9588-389ecdafe353-kube-api-access-fhqmx\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.579360 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.579336    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8985a09b-1f37-4792-9588-389ecdafe353-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.581856 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.581832    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.582714 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.582683    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.589506 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.589476    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqmx\" (UniqueName: \"kubernetes.io/projected/8985a09b-1f37-4792-9588-389ecdafe353-kube-api-access-fhqmx\") pod \"maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:50.762772 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:50.762686    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:51.068952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:51.068898    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"]
Nov 03 20:20:51.842474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:51.842356    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt" event={"ID":"8985a09b-1f37-4792-9588-389ecdafe353","Type":"ContainerStarted","Data":"bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94"}
Nov 03 20:20:51.842474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:51.842423    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt" event={"ID":"8985a09b-1f37-4792-9588-389ecdafe353","Type":"ContainerStarted","Data":"a7d3f0a3058aaf019db8b073b45a4342236bb5e92d29c900aff5786d963080ad"}
Nov 03 20:20:51.842912 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:51.842495    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:51.859160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:51.859100    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt" podStartSLOduration=1.3789964160000001 podStartE2EDuration="1.859082727s" totalImagesPullingTime="480.086311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:51.858242347 +0000 UTC m=+2434.371064130" watchObservedRunningTime="2026-11-03 20:20:51.859082727 +0000 UTC m=+2434.371904511"
Nov 03 20:20:53.380516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:53.380428    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:53.398720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:53.398689    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:53.870854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:53.870817    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:20:54.705054 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:54.705016    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"]
Nov 03 20:20:54.866006 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:54.865963    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" podUID="73112745-0b21-4933-856a-0830555ca2a3" containerName="istio-proxy" containerID="cri-o://c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5" gracePeriod=30
Nov 03 20:20:57.856149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:20:57.856072    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:20:58.798887 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:20:58.798848    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:21:00.170730 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.170694    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:21:00.276371 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276331    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-kube-api-access-7wjcf\" (UniqueName: \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-kube-api-access-7wjcf\") pod \"73112745-0b21-4933-856a-0830555ca2a3\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") "
Nov 03 20:21:00.276595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276388    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-socket\") pod \"73112745-0b21-4933-856a-0830555ca2a3\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") "
Nov 03 20:21:00.276595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276418    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-envoy\") pod \"73112745-0b21-4933-856a-0830555ca2a3\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") "
Nov 03 20:21:00.276595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276511    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73112745-0b21-4933-856a-0830555ca2a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73112745-0b21-4933-856a-0830555ca2a3-istiod-ca-cert\") pod \"73112745-0b21-4933-856a-0830555ca2a3\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") "
Nov 03 20:21:00.276595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276537    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-istio-token\" (UniqueName: \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-istio-token\") pod \"73112745-0b21-4933-856a-0830555ca2a3\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") "
Nov 03 20:21:00.276825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276733    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-certs\") pod \"73112745-0b21-4933-856a-0830555ca2a3\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") "
Nov 03 20:21:00.276825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276712    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-socket" pod "73112745-0b21-4933-856a-0830555ca2a3" (UID: "73112745-0b21-4933-856a-0830555ca2a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:00.276825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276777    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-data\") pod \"73112745-0b21-4933-856a-0830555ca2a3\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") "
Nov 03 20:21:00.277035 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276850    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73112745-0b21-4933-856a-0830555ca2a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73112745-0b21-4933-856a-0830555ca2a3-istio-podinfo\") pod \"73112745-0b21-4933-856a-0830555ca2a3\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") "
Nov 03 20:21:00.277035 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276907    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-credential-socket\") pod \"73112745-0b21-4933-856a-0830555ca2a3\" (UID: \"73112745-0b21-4933-856a-0830555ca2a3\") "
Nov 03 20:21:00.277035 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276963    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-certs" pod "73112745-0b21-4933-856a-0830555ca2a3" (UID: "73112745-0b21-4933-856a-0830555ca2a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:00.277035 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.276897    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73112745-0b21-4933-856a-0830555ca2a3-istiod-ca-cert" pod "73112745-0b21-4933-856a-0830555ca2a3" (UID: "73112745-0b21-4933-856a-0830555ca2a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:00.277224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.277048    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-data" pod "73112745-0b21-4933-856a-0830555ca2a3" (UID: "73112745-0b21-4933-856a-0830555ca2a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:00.277376 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.277286    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-credential-socket" pod "73112745-0b21-4933-856a-0830555ca2a3" (UID: "73112745-0b21-4933-856a-0830555ca2a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:00.277613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.277592    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:00.277695 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.277617    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:00.277695 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.277632    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:00.277695 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.277645    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:00.277695 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.277658    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73112745-0b21-4933-856a-0830555ca2a3-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:00.279002 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.278975    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-istio-token" pod "73112745-0b21-4933-856a-0830555ca2a3" (UID: "73112745-0b21-4933-856a-0830555ca2a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:00.279182 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.279153    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73112745-0b21-4933-856a-0830555ca2a3-istio-podinfo" pod "73112745-0b21-4933-856a-0830555ca2a3" (UID: "73112745-0b21-4933-856a-0830555ca2a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:21:00.279280 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.279221    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-kube-api-access-7wjcf" pod "73112745-0b21-4933-856a-0830555ca2a3" (UID: "73112745-0b21-4933-856a-0830555ca2a3"). InnerVolumeSpecName "kube-api-access-7wjcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:00.279280 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.279269    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-envoy" pod "73112745-0b21-4933-856a-0830555ca2a3" (UID: "73112745-0b21-4933-856a-0830555ca2a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:00.379482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.379447    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73112745-0b21-4933-856a-0830555ca2a3-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:00.379482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.379483    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wjcf\" (UniqueName: \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-kube-api-access-7wjcf\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:00.379704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.379498    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73112745-0b21-4933-856a-0830555ca2a3-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:00.379704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.379510    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73112745-0b21-4933-856a-0830555ca2a3-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:00.934822 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.934786    5600 generic.go:356] "Generic (PLEG): container finished" podID="73112745-0b21-4933-856a-0830555ca2a3" containerID="c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5" exitCode=0
Nov 03 20:21:00.935016 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.934846    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"
Nov 03 20:21:00.935016 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.934855    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" event={"ID":"73112745-0b21-4933-856a-0830555ca2a3","Type":"ContainerDied","Data":"c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5"}
Nov 03 20:21:00.935016 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.934909    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk" event={"ID":"73112745-0b21-4933-856a-0830555ca2a3","Type":"ContainerDied","Data":"5ee559d925100fdfc48b1f2fcf260f95fadb299a97cdfa2749d822f075895629"}
Nov 03 20:21:00.935016 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.934931    5600 scope.go:122] "RemoveContainer" containerID="c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5"
Nov 03 20:21:00.947817 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.947789    5600 scope.go:122] "RemoveContainer" containerID="c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5"
Nov 03 20:21:00.948166 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:21:00.948144    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5\": container with ID starting with c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5 not found: ID does not exist" containerID="c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5"
Nov 03 20:21:00.948264 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.948192    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5"} err="failed to get container status \"c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5\": rpc error: code = NotFound desc = could not find container \"c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5\": container with ID starting with c71611618b7893926031ca7452ce09359224e0c7b677ed7e1612e07f695c3fc5 not found: ID does not exist"
Nov 03 20:21:00.963291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.963257    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"]
Nov 03 20:21:00.968000 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:00.967966    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-kgjrk"]
Nov 03 20:21:01.618519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.618487    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73112745-0b21-4933-856a-0830555ca2a3" path="/var/lib/kubelet/pods/73112745-0b21-4933-856a-0830555ca2a3/volumes"
Nov 03 20:21:01.834693 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.834659    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"]
Nov 03 20:21:01.835288 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.835274    5600 cpu_manager.go:414] "containerMap: removing container" podUID="73112745-0b21-4933-856a-0830555ca2a3" containerName="istio-proxy"
Nov 03 20:21:01.835365 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.835289    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73112745-0b21-4933-856a-0830555ca2a3" containerName="istio-proxy"
Nov 03 20:21:01.835426 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.835416    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="73112745-0b21-4933-856a-0830555ca2a3" containerName="istio-proxy"
Nov 03 20:21:01.840795 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.840768    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:01.849940 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.849913    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-456f55-serving-cert\""
Nov 03 20:21:01.871149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.871063    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"]
Nov 03 20:21:01.898550 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.898519    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:01.898696 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.898564    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:01.898696 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.898648    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:01.898936 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:01.898912    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcjz\" (UniqueName: \"kubernetes.io/projected/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-kube-api-access-mhcjz\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:02.000148 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:02.000107    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcjz\" (UniqueName: \"kubernetes.io/projected/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-kube-api-access-mhcjz\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:02.000370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:02.000245    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:02.000370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:02.000283    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:02.000370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:02.000326    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:02.001090 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:21:02.001061    5600 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-serving-cert: secret "maas-api-e2e-shared-a-w2-456f55-serving-cert" not found
Nov 03 20:21:02.001203 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:02.001100    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:02.001203 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:21:02.001156    5600 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-maas-api-tls podName:b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd nodeName:}" failed. No retries permitted until 2026-11-03 20:21:02.501136665 +0000 UTC m=+2445.013958440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-maas-api-tls") pod "maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz" (UID: "b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd") : secret "maas-api-e2e-shared-a-w2-456f55-serving-cert" not found
Nov 03 20:21:02.004091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:02.004062    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:02.008618 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:02.008592    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcjz\" (UniqueName: \"kubernetes.io/projected/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-kube-api-access-mhcjz\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:02.517460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:02.511673    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:02.521454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:02.519094    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:02.773508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:02.766751    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:03.169974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:03.169941    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"]
Nov 03 20:21:03.171913 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:21:03.171881    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1df3f04_e3c8_4aae_b58c_e8a9ab2bb2fd.slice/crio-6562f2b518e2aec4c40f8da4f9938128697cbbfd07d937e0776ac7194a3adc5e WatchSource:0}: Error finding container 6562f2b518e2aec4c40f8da4f9938128697cbbfd07d937e0776ac7194a3adc5e: Status 404 returned error can't find the container with id 6562f2b518e2aec4c40f8da4f9938128697cbbfd07d937e0776ac7194a3adc5e
Nov 03 20:21:03.978747 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:03.978702    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz" event={"ID":"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd","Type":"ContainerStarted","Data":"d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8"}
Nov 03 20:21:03.979199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:03.978771    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz" event={"ID":"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd","Type":"ContainerStarted","Data":"6562f2b518e2aec4c40f8da4f9938128697cbbfd07d937e0776ac7194a3adc5e"}
Nov 03 20:21:03.979199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:03.978801    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:03.998701 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:03.998642    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz" podStartSLOduration=2.5661961079999998 podStartE2EDuration="2.998626985s" totalImagesPullingTime="432.430877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:03.995482782 +0000 UTC m=+2446.508304567" watchObservedRunningTime="2026-11-03 20:21:03.998626985 +0000 UTC m=+2446.511448768"
Nov 03 20:21:09.994681 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:09.994646    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:21:19.202702 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.201915    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"]
Nov 03 20:21:19.208357 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.208324    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.223756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.223711    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"]
Nov 03 20:21:19.303879 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.303845    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-ee2323-openshift-default-dockercfg-krfdd\""
Nov 03 20:21:19.304417 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.304388    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.304551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.304428    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.304551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.304505    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.304671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.304603    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.304671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.304638    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.304790 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.304716    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.304790 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.304769    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.304899 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.304888    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.304954 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.304932    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2pv8\" (UniqueName: \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-kube-api-access-p2pv8\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.406996 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.406949    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.407216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.407031    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2pv8\" (UniqueName: \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-kube-api-access-p2pv8\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.407216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.407081    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.407216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.407106    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.407375 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.407258    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.407375 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.407366    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.407537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.407405    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.407537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.407502    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.407641 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.407584    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.407641 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.407590    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.408249 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.408221    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.408580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.408507    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.408580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.408602    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.408580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.408614    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.410847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.410823    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.411550 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.411519    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.416384 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.416359    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.416736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.416715    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2pv8\" (UniqueName: \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-kube-api-access-p2pv8\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.547545 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.547444    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:19.908519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.902775    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"]
Nov 03 20:21:19.960202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.960131    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:19.960612 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.960505    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:19.960612 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:19.960574    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:20.132562 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:20.132520    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" event={"ID":"1b05c7a1-d6cf-4963-a997-62c39ca1217b","Type":"ContainerStarted","Data":"eac60e97736482836a2a667c6dae24d1944be3d401d1cef1a6a4c307cd4580c4"}
Nov 03 20:21:20.132729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:20.132591    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" event={"ID":"1b05c7a1-d6cf-4963-a997-62c39ca1217b","Type":"ContainerStarted","Data":"9a9bbeeefd1062db4dd76e310c1d802af9ed550f1fadc591c191aa957662f501"}
Nov 03 20:21:20.155588 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:20.155478    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" podStartSLOduration=1.155456435 podStartE2EDuration="1.155456435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:20.152378823 +0000 UTC m=+2462.665200608" watchObservedRunningTime="2026-11-03 20:21:20.155456435 +0000 UTC m=+2462.668278220"
Nov 03 20:21:20.548602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:20.548558    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:21.459022 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:21.458991    5600 scope.go:122] "RemoveContainer" containerID="5157ed0588b6bb223360454f3c1c24d8500d866bd4f0a2193dc0fff35fe66f95"
Nov 03 20:21:21.550414 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:21.550386    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:21:21.550754 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:21.550457    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:21:22.548508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:22.548468    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:21:22.548732 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:22.548537    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:21:23.548152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:23.548114    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:21:23.548625 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:23.548181    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:21:24.548676 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:24.548638    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:21:24.549159 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:24.548710    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:21:25.556463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:25.547853    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:21:25.556463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:25.547916    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:21:26.548370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:26.547973    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:21:26.548370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:26.548043    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:21:27.553491 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:27.553423    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:27.553988 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:27.553968    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:27.554958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:27.554938    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:21:28.800722 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:21:28.800681    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:21:34.221508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.221465    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"]
Nov 03 20:21:34.229717 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.229686    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.243470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.243420    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"]
Nov 03 20:21:34.315090 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.315045    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqwwk\" (UniqueName: \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-kube-api-access-rqwwk\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.315390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.315371    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5409036e-08df-4aa4-8245-23575462a0be-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.315645 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.315627    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.315779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.315765    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5409036e-08df-4aa4-8245-23575462a0be-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.315919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.315904    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.316032 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.316020    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.316233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.316219    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.316331 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.316316    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.316487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.316473    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.417968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.417927    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.417968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.417972    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.418230 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.418018    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.418395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.418277    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqwwk\" (UniqueName: \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-kube-api-access-rqwwk\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.418518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.418491    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5409036e-08df-4aa4-8245-23575462a0be-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.418597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.418538    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.418597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.418579    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5409036e-08df-4aa4-8245-23575462a0be-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.418597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.418580    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.418769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.418672    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.418769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.418745    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.419012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.418951    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.419012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.419012    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.419012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.419036    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.419343 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.419315    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5409036e-08df-4aa4-8245-23575462a0be-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.420815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.420790    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.421726 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.421703    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5409036e-08df-4aa4-8245-23575462a0be-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.427824 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.427794    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.428001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.427980    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqwwk\" (UniqueName: \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-kube-api-access-rqwwk\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.573341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.573258    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:34.659211 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.659153    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"]
Nov 03 20:21:34.666925 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.666894    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.682070 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.680349    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-a8e19e-serving-cert\""
Nov 03 20:21:34.698989 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.698948    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"]
Nov 03 20:21:34.841781 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.836419    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.841781 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.836614    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.841781 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.836664    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/011eb7d6-11dc-4fa6-8bfa-82897107b779-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.841781 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.836861    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4njs\" (UniqueName: \"kubernetes.io/projected/011eb7d6-11dc-4fa6-8bfa-82897107b779-kube-api-access-n4njs\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.883915 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:21:34.882858    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5409036e_08df_4aa4_8245_23575462a0be.slice/crio-46ccf309b63d87280ee2ca33eb2bb3e7ef429e588c8d8b235b7cd831ee93968b WatchSource:0}: Error finding container 46ccf309b63d87280ee2ca33eb2bb3e7ef429e588c8d8b235b7cd831ee93968b: Status 404 returned error can't find the container with id 46ccf309b63d87280ee2ca33eb2bb3e7ef429e588c8d8b235b7cd831ee93968b
Nov 03 20:21:34.887846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.887803    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:34.887974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.887892    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:34.887974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.887941    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:34.894234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.894201    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"]
Nov 03 20:21:34.938168 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.938123    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.938325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.938198    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/011eb7d6-11dc-4fa6-8bfa-82897107b779-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.938325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.938321    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4njs\" (UniqueName: \"kubernetes.io/projected/011eb7d6-11dc-4fa6-8bfa-82897107b779-kube-api-access-n4njs\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.938493 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.938453    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.939261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.939233    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/011eb7d6-11dc-4fa6-8bfa-82897107b779-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.941168 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.941145    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.941918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.941886    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:34.947250 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:34.947220    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4njs\" (UniqueName: \"kubernetes.io/projected/011eb7d6-11dc-4fa6-8bfa-82897107b779-kube-api-access-n4njs\") pod \"maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:35.009093 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:35.009056    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:35.328334 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:35.328224    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" event={"ID":"5409036e-08df-4aa4-8245-23575462a0be","Type":"ContainerStarted","Data":"1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48"}
Nov 03 20:21:35.328334 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:35.328291    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" event={"ID":"5409036e-08df-4aa4-8245-23575462a0be","Type":"ContainerStarted","Data":"46ccf309b63d87280ee2ca33eb2bb3e7ef429e588c8d8b235b7cd831ee93968b"}
Nov 03 20:21:35.332603 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:35.332573    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"]
Nov 03 20:21:35.336612 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:21:35.336574    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011eb7d6_11dc_4fa6_8bfa_82897107b779.slice/crio-8e521cc1727be8a3b3f3ac25bf0b374acfdd61b95c414db10c18bdc206c704bc WatchSource:0}: Error finding container 8e521cc1727be8a3b3f3ac25bf0b374acfdd61b95c414db10c18bdc206c704bc: Status 404 returned error can't find the container with id 8e521cc1727be8a3b3f3ac25bf0b374acfdd61b95c414db10c18bdc206c704bc
Nov 03 20:21:35.355230 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:35.352308    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" podStartSLOduration=1.352286053 podStartE2EDuration="1.352286053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:35.348705622 +0000 UTC m=+2477.861527406" watchObservedRunningTime="2026-11-03 20:21:35.352286053 +0000 UTC m=+2477.865107831"
Nov 03 20:21:35.578052 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:35.577837    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:21:35.578052 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:35.577919    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:21:35.578379 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:35.575955    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:36.341889 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:36.341844    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l" event={"ID":"011eb7d6-11dc-4fa6-8bfa-82897107b779","Type":"ContainerStarted","Data":"a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225"}
Nov 03 20:21:36.342298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:36.341948    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:36.342298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:36.341975    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l" event={"ID":"011eb7d6-11dc-4fa6-8bfa-82897107b779","Type":"ContainerStarted","Data":"8e521cc1727be8a3b3f3ac25bf0b374acfdd61b95c414db10c18bdc206c704bc"}
Nov 03 20:21:36.361147 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:36.361091    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l" podStartSLOduration=1.876187422 podStartE2EDuration="2.361076027s" totalImagesPullingTime="484.888605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:36.357763124 +0000 UTC m=+2478.870584908" watchObservedRunningTime="2026-11-03 20:21:36.361076027 +0000 UTC m=+2478.873897810"
Nov 03 20:21:36.574179 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:36.574143    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:21:36.574390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:36.574219    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:21:37.574084 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:37.574050    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:21:37.574579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:37.574100    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:21:38.574262 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:38.574223    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:21:38.574917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:38.574883    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:21:39.277252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.277209    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"]
Nov 03 20:21:39.297283 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.289778    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.353565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.353475    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"]
Nov 03 20:21:39.359636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.359595    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.359636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.359646    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.359896 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.359675    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.359896 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.359729    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4kkb\" (UniqueName: \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-kube-api-access-n4kkb\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.359896 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.359765    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.359896 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.359787    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.360129 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.359973    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.360129 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.360070    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139b0032-cd8f-426e-bdee-77ebb7955b50-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.360129 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.360102    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.375933 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.375898    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a476f8-openshift-default-dockercfg-lsx9m\""
Nov 03 20:21:39.461912 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.461873    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.461977    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139b0032-cd8f-426e-bdee-77ebb7955b50-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462008    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462098    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462128    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462160    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462222    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4kkb\" (UniqueName: \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-kube-api-access-n4kkb\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462255    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462281    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462336    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462466    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462507    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462649    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.462756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.462649    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139b0032-cd8f-426e-bdee-77ebb7955b50-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.464785 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.464761    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.465079 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.465061    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.470340 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.470308    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4kkb\" (UniqueName: \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-kube-api-access-n4kkb\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.470609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.470583    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.573765 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.573687    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:21:39.573765 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.573737    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:21:39.641070 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.641027    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:39.967634 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.967602    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"]
Nov 03 20:21:39.969920 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:21:39.969229    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139b0032_cd8f_426e_bdee_77ebb7955b50.slice/crio-b3b640624b87ee729fd0ecc32345cbbee75ac648325686a6edfdfa2381fe7818 WatchSource:0}: Error finding container b3b640624b87ee729fd0ecc32345cbbee75ac648325686a6edfdfa2381fe7818: Status 404 returned error can't find the container with id b3b640624b87ee729fd0ecc32345cbbee75ac648325686a6edfdfa2381fe7818
Nov 03 20:21:39.972426 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.972396    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:39.972615 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.972600    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:39.972709 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:39.972698    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:40.375713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:40.375617    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" event={"ID":"139b0032-cd8f-426e-bdee-77ebb7955b50","Type":"ContainerStarted","Data":"97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4"}
Nov 03 20:21:40.375713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:40.375681    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" event={"ID":"139b0032-cd8f-426e-bdee-77ebb7955b50","Type":"ContainerStarted","Data":"b3b640624b87ee729fd0ecc32345cbbee75ac648325686a6edfdfa2381fe7818"}
Nov 03 20:21:40.428457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:40.414351    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" podStartSLOduration=1.414327704 podStartE2EDuration="1.414327704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:40.408197826 +0000 UTC m=+2482.921019608" watchObservedRunningTime="2026-11-03 20:21:40.414327704 +0000 UTC m=+2482.927149488"
Nov 03 20:21:40.574055 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:40.574013    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:21:40.574252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:40.574085    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:21:40.644412 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:40.642552    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:41.573958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:41.573927    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:21:41.574129 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:41.573983    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:21:41.644916 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:41.644882    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:21:41.645478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:41.644939    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:21:42.355883 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:42.355854    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:21:42.580715 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:42.580682    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:42.581251 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:42.581228    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:42.601461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:42.586846    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:21:42.643920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:42.643883    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:21:42.644122 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:42.643946    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:21:43.326158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.326111    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"]
Nov 03 20:21:43.341904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.341515    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.352424 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.351077    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"]
Nov 03 20:21:43.352424 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.351932    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-b55105-serving-cert\""
Nov 03 20:21:43.388461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.372678    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k622n\" (UniqueName: \"kubernetes.io/projected/d8868e28-6951-4711-88ba-498cf360042e-kube-api-access-k622n\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.388461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.372819    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.388461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.372892    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.388461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.372989    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8868e28-6951-4711-88ba-498cf360042e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.474098 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.474057    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.474325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.474193    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.474325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.474247    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8868e28-6951-4711-88ba-498cf360042e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.474472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.474422    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k622n\" (UniqueName: \"kubernetes.io/projected/d8868e28-6951-4711-88ba-498cf360042e-kube-api-access-k622n\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.475640 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.475610    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8868e28-6951-4711-88ba-498cf360042e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.477282 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.477259    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.477399 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.477323    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.529621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.529515    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k622n\" (UniqueName: \"kubernetes.io/projected/d8868e28-6951-4711-88ba-498cf360042e-kube-api-access-k622n\") pod \"maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:43.647948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.647909    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:21:43.650858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.650812    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:21:43.678342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:43.678307    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:44.342857 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:21:44.342822    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8868e28_6951_4711_88ba_498cf360042e.slice/crio-ee548cf036c59d1dfe1223dc27be63e2b799f09120c7d92b7ee5bc80fd24233e WatchSource:0}: Error finding container ee548cf036c59d1dfe1223dc27be63e2b799f09120c7d92b7ee5bc80fd24233e: Status 404 returned error can't find the container with id ee548cf036c59d1dfe1223dc27be63e2b799f09120c7d92b7ee5bc80fd24233e
Nov 03 20:21:44.346083 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:44.346051    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"]
Nov 03 20:21:44.649537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:44.632033    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp" event={"ID":"d8868e28-6951-4711-88ba-498cf360042e","Type":"ContainerStarted","Data":"ee548cf036c59d1dfe1223dc27be63e2b799f09120c7d92b7ee5bc80fd24233e"}
Nov 03 20:21:44.665612 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:44.665569    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:21:44.665810 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:44.665620    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:21:45.641531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:45.641488    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:21:45.641981 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:45.641552    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:21:45.642871 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:45.642826    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp" event={"ID":"d8868e28-6951-4711-88ba-498cf360042e","Type":"ContainerStarted","Data":"272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b"}
Nov 03 20:21:45.642994 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:45.642899    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:46.641664 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:46.641621    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:21:46.642182 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:46.641687    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:21:47.646653 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:47.646624    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:47.647071 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:47.647058    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:47.647858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:47.647839    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:21:47.670188 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:47.670129    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp" podStartSLOduration=4.068161343 podStartE2EDuration="4.670115241s" totalImagesPullingTime="601.953898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:45.663274959 +0000 UTC m=+2488.176096742" watchObservedRunningTime="2026-11-03 20:21:47.670115241 +0000 UTC m=+2490.182937023"
Nov 03 20:21:49.233941 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.233867    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"]
Nov 03 20:21:49.238877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.238854    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.251968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.251943    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"]
Nov 03 20:21:49.319583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.319534    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9ln\" (UniqueName: \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-kube-api-access-5w9ln\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.319769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.319605    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.319769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.319728    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.319928 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.319905    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.319970 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.319944    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.320047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.320033    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.320083 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.320062    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.320116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.320084    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f8d3e0-be03-463c-92a4-27903498fe1e-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.320234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.320222    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.421352 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.421317    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.421590 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.421375    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.421590 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.421424    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.421590 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.421477    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.421590 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.421502    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f8d3e0-be03-463c-92a4-27903498fe1e-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.421839 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.421606    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.421839 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.421639    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9ln\" (UniqueName: \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-kube-api-access-5w9ln\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.421839 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.421706    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.421839 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.421773    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.421839 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.421798    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.422112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.421886    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.422112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.422101    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.422216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.422195    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.422581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.422560    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f8d3e0-be03-463c-92a4-27903498fe1e-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.423925 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.423901    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.424259 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.424241    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.438194 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.438161    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.438325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.438268    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9ln\" (UniqueName: \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-kube-api-access-5w9ln\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.559894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.559804    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:49.894386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.894358    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"]
Nov 03 20:21:49.895803 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:21:49.895779    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f8d3e0_be03_463c_92a4_27903498fe1e.slice/crio-5b5f84b8b096edcc79414d991afabd8c8174732050d27bda5034e783b401b3f2 WatchSource:0}: Error finding container 5b5f84b8b096edcc79414d991afabd8c8174732050d27bda5034e783b401b3f2: Status 404 returned error can't find the container with id 5b5f84b8b096edcc79414d991afabd8c8174732050d27bda5034e783b401b3f2
Nov 03 20:21:49.899251 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.899223    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:49.899317 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.899286    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:49.899361 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:49.899316    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:50.711909 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:50.711870    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" event={"ID":"19f8d3e0-be03-463c-92a4-27903498fe1e","Type":"ContainerStarted","Data":"89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d"}
Nov 03 20:21:50.712593 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:50.712566    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" event={"ID":"19f8d3e0-be03-463c-92a4-27903498fe1e","Type":"ContainerStarted","Data":"5b5f84b8b096edcc79414d991afabd8c8174732050d27bda5034e783b401b3f2"}
Nov 03 20:21:50.736091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:50.736038    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" podStartSLOduration=1.7360183710000001 podStartE2EDuration="1.736018371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:50.732495853 +0000 UTC m=+2493.245317640" watchObservedRunningTime="2026-11-03 20:21:50.736018371 +0000 UTC m=+2493.248840156"
Nov 03 20:21:51.561036 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:51.560998    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:51.562623 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:51.562595    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:21:51.562759 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:51.562646    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" podUID="19f8d3e0-be03-463c-92a4-27903498fe1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:21:51.659230 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:51.659194    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:21:52.560847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:52.560818    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:21:52.561247 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:52.560870    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" podUID="19f8d3e0-be03-463c-92a4-27903498fe1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:21:53.562256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:53.562223    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:21:53.562940 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:53.562906    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" podUID="19f8d3e0-be03-463c-92a4-27903498fe1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:21:54.324124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.319557    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"]
Nov 03 20:21:54.332048 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.329417    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"]
Nov 03 20:21:54.332048 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.329587    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.439493 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.439453    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.439685 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.439508    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.439685 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.439579    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.439685 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.439642    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.439685 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.439665    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.439685 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.439687    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.439888 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.439826    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.439941 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.439925    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm7z9\" (UniqueName: \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-kube-api-access-cm7z9\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.440082 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.440070    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542021 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.541983    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542245 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542080    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm7z9\" (UniqueName: \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-kube-api-access-cm7z9\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542245 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542230    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542381 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542271    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542381 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542314    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542381 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542350    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542417    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542495    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542530    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542689    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542719    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542814    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.542846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.542839    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.543246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.543175    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.545574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.545549    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.546228 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.546204    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.550547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.550519    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.550946 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.550927    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm7z9\" (UniqueName: \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-kube-api-access-cm7z9\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:54.561021 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.560997    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:21:54.561126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.561058    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" podUID="19f8d3e0-be03-463c-92a4-27903498fe1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:21:54.655927 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:54.655895    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:55.018565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:55.018534    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"]
Nov 03 20:21:55.021567 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:21:55.021532    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba674abb_7d11_4a99_b3a1_e630b6d41a48.slice/crio-4f19ffe4ac8f85f0ef46da6cdcfaf1b07e14f803a89e7f69304f28ace5529583 WatchSource:0}: Error finding container 4f19ffe4ac8f85f0ef46da6cdcfaf1b07e14f803a89e7f69304f28ace5529583: Status 404 returned error can't find the container with id 4f19ffe4ac8f85f0ef46da6cdcfaf1b07e14f803a89e7f69304f28ace5529583
Nov 03 20:21:55.025096 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:55.025063    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:55.025202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:55.025136    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:55.025202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:55.025169    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:21:55.560913 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:55.560884    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:21:55.561060 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:55.560935    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" podUID="19f8d3e0-be03-463c-92a4-27903498fe1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:21:55.771815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:55.771772    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" event={"ID":"ba674abb-7d11-4a99-b3a1-e630b6d41a48","Type":"ContainerStarted","Data":"c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02"}
Nov 03 20:21:55.772185 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:55.771844    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" event={"ID":"ba674abb-7d11-4a99-b3a1-e630b6d41a48","Type":"ContainerStarted","Data":"4f19ffe4ac8f85f0ef46da6cdcfaf1b07e14f803a89e7f69304f28ace5529583"}
Nov 03 20:21:55.814520 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:55.813625    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" podStartSLOduration=1.8136010009999999 podStartE2EDuration="1.813601001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:55.802989298 +0000 UTC m=+2498.315811102" watchObservedRunningTime="2026-11-03 20:21:55.813601001 +0000 UTC m=+2498.326422784"
Nov 03 20:21:56.656710 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:56.656667    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:21:56.658961 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:56.658424    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:21:56.658961 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:56.658521    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" podUID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 20:21:56.801460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:56.800583    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:56.802053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:56.802032    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:56.813480 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:56.811739    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:21:57.656602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:57.656565    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:21:57.656798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:57.656629    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" podUID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 20:21:58.657065 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:58.657029    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:21:58.657509 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:58.657091    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" podUID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 20:21:58.839677 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:21:58.839603    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:21:59.656370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:59.656310    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:21:59.656748 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:21:59.656387    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" podUID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 20:22:00.657142 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:00.657109    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:22:00.657659 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:00.657172    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" podUID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 20:22:01.861256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:01.861222    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:22:01.863466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:01.863409    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:22:01.865988 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:01.865962    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:22:18.861668 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.861628    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"]
Nov 03 20:22:18.872944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.872905    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"]
Nov 03 20:22:18.879567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.873017    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:18.880025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.879991    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a8e19e\"/\"default-dockercfg-pj8hq\""
Nov 03 20:22:18.880181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.880071    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a8e19e\"/\"auth-test-model-a8e19e-kserve-self-signed-certs\""
Nov 03 20:22:18.880181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.880118    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a8e19e\"/\"kube-root-ca.crt\""
Nov 03 20:22:18.880181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.880154    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a8e19e\"/\"openshift-service-ca.crt\""
Nov 03 20:22:18.925923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.925887    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-dshm\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:18.926111 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.925934    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92ae809-042d-4ddf-bd1d-f582025bea07-tls-certs\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:18.926190 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.926142    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-tmp-dir\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:18.926252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.926203    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-home\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:18.926346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.926328    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2pkt\" (UniqueName: \"kubernetes.io/projected/d92ae809-042d-4ddf-bd1d-f582025bea07-kube-api-access-p2pkt\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:18.926495 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:18.926481    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-model-cache\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.043458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.042569    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-home\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.043458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.042720    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2pkt\" (UniqueName: \"kubernetes.io/projected/d92ae809-042d-4ddf-bd1d-f582025bea07-kube-api-access-p2pkt\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.043458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.042870    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-model-cache\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.043458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.042927    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-dshm\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.043458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.042958    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92ae809-042d-4ddf-bd1d-f582025bea07-tls-certs\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.043458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.043099    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-tmp-dir\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.043955 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.043596    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-tmp-dir\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.043955 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.043890    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-home\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.063523 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.052753    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-model-cache\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.063523 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.061325    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92ae809-042d-4ddf-bd1d-f582025bea07-tls-certs\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.083463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.069719    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-dshm\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.083463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.079988    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2pkt\" (UniqueName: \"kubernetes.io/projected/d92ae809-042d-4ddf-bd1d-f582025bea07-kube-api-access-p2pkt\") pod \"auth-test-model-a8e19e-kserve-66646678bf-jlqxl\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") " pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.211821 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.211785    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:19.527064 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:19.527029    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"]
Nov 03 20:22:19.528179 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:22:19.528146    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd92ae809_042d_4ddf_bd1d_f582025bea07.slice/crio-5ced858db382bb6941aa3b890ae31820ce913f86b28db4968afcf3c87592a0c8 WatchSource:0}: Error finding container 5ced858db382bb6941aa3b890ae31820ce913f86b28db4968afcf3c87592a0c8: Status 404 returned error can't find the container with id 5ced858db382bb6941aa3b890ae31820ce913f86b28db4968afcf3c87592a0c8
Nov 03 20:22:20.133355 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:20.133314    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl" event={"ID":"d92ae809-042d-4ddf-bd1d-f582025bea07","Type":"ContainerStarted","Data":"15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126"}
Nov 03 20:22:20.133923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:20.133371    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl" event={"ID":"d92ae809-042d-4ddf-bd1d-f582025bea07","Type":"ContainerStarted","Data":"5ced858db382bb6941aa3b890ae31820ce913f86b28db4968afcf3c87592a0c8"}
Nov 03 20:22:20.154471 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:20.154388    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl" podStartSLOduration=2.154371088 podStartE2EDuration="2.154371088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:20.153967322 +0000 UTC m=+2522.666789106" watchObservedRunningTime="2026-11-03 20:22:20.154371088 +0000 UTC m=+2522.667192879"
Nov 03 20:22:20.212497 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:20.212468    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:21.560514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:21.560473    5600 scope.go:122] "RemoveContainer" containerID="49310ea5c57dffc52537c89cde5d4e80bb638e7323108623737b7646c662ac41"
Nov 03 20:22:28.806078 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:22:28.805982    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:22:29.213062 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:29.213026    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:29.227713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:29.227683    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:30.228846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:30.228816    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:22:32.409092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.409052    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"]
Nov 03 20:22:32.415169 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.415133    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.441781 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.441748    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b55105\"/\"openshift-service-ca.crt\""
Nov 03 20:22:32.442124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.442102    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b55105\"/\"default-dockercfg-5rqhs\""
Nov 03 20:22:32.442354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.442335    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b55105\"/\"kube-root-ca.crt\""
Nov 03 20:22:32.442494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.442359    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b55105\"/\"auth-test-model-b55105-kserve-self-signed-certs\""
Nov 03 20:22:32.447647 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.447624    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"]
Nov 03 20:22:32.501391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.501351    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-dshm\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.501627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.501493    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tmp-dir\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.501627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.501597    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-home\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.501861 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.501837    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tls-certs\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.501934 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.501902    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq59m\" (UniqueName: \"kubernetes.io/projected/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-kube-api-access-lq59m\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.501998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.501937    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-model-cache\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.625456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.620865    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-dshm\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.625456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.621562    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tmp-dir\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.625456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.621645    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-home\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.625456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.621992    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tls-certs\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.625456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.622040    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq59m\" (UniqueName: \"kubernetes.io/projected/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-kube-api-access-lq59m\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.625456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.622117    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-model-cache\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.625456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.622779    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-model-cache\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.625456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.623167    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tmp-dir\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.625456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.623478    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-home\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.632943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.630915    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-dshm\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.641453 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.636925    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq59m\" (UniqueName: \"kubernetes.io/projected/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-kube-api-access-lq59m\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.675569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.667089    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tls-certs\") pod \"auth-test-model-b55105-kserve-55f544647c-qxlbm\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") " pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:32.772458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:32.771416    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:33.090986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:33.090951    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"]
Nov 03 20:22:33.092213 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:22:33.092185    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb286a49_946b_4a5f_a2c8_6d116a51eb0b.slice/crio-6fb6bad49dcbae401580302069b5afd3c33166aec0a34bff8122b7f8acdefe41 WatchSource:0}: Error finding container 6fb6bad49dcbae401580302069b5afd3c33166aec0a34bff8122b7f8acdefe41: Status 404 returned error can't find the container with id 6fb6bad49dcbae401580302069b5afd3c33166aec0a34bff8122b7f8acdefe41
Nov 03 20:22:33.241171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:33.241130    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm" event={"ID":"fb286a49-946b-4a5f-a2c8-6d116a51eb0b","Type":"ContainerStarted","Data":"4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654"}
Nov 03 20:22:33.241375 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:33.241195    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm" event={"ID":"fb286a49-946b-4a5f-a2c8-6d116a51eb0b","Type":"ContainerStarted","Data":"6fb6bad49dcbae401580302069b5afd3c33166aec0a34bff8122b7f8acdefe41"}
Nov 03 20:22:33.261737 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:33.261688    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm" podStartSLOduration=1.26167481 podStartE2EDuration="1.26167481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:33.257206512 +0000 UTC m=+2535.770028286" watchObservedRunningTime="2026-11-03 20:22:33.26167481 +0000 UTC m=+2535.774496639"
Nov 03 20:22:33.772183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:33.772152    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:34.662805 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.662771    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"]
Nov 03 20:22:34.667373 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.667347    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.675780 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.675750    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-46c34c-openshift-default-dockercfg-55bnp\""
Nov 03 20:22:34.709504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.705896    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"]
Nov 03 20:22:34.767714 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.767665    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.767910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.767737    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.767910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.767840    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.768041 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.767932    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.768111 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.768044    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.768162 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.768142    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.768288 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.768272    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkmj\" (UniqueName: \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-kube-api-access-gzkmj\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.768352 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.768309    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.768574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.768552    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.870013 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.869968    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.870575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870035    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.870575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870080    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.870575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870116    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.870575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870204    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.870575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870274    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.870575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870396    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkmj\" (UniqueName: \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-kube-api-access-gzkmj\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.870575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870409    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.870575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870453    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.870575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870517    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.871047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870598    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.871047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870609    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.871047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870733    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.871047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.870914    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.873124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.873095    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.873351 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.873328    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.881250 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.881217    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkmj\" (UniqueName: \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-kube-api-access-gzkmj\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:34.881385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:34.881285    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:35.015319 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:35.015207    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:35.311867 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:22:35.311833    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a71e36d_e50f_4c6e_ba07_fa2fb1cc2ee7.slice/crio-7e2a308f8de53b1d1eceae704f3c4a26a0237c44d014a59bbec9e91da9350c79 WatchSource:0}: Error finding container 7e2a308f8de53b1d1eceae704f3c4a26a0237c44d014a59bbec9e91da9350c79: Status 404 returned error can't find the container with id 7e2a308f8de53b1d1eceae704f3c4a26a0237c44d014a59bbec9e91da9350c79
Nov 03 20:22:35.312291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:35.312235    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"]
Nov 03 20:22:35.315517 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:35.315478    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:22:35.315638 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:35.315567    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:22:35.315638 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:35.315602    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:22:36.267646 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:36.267609    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g" event={"ID":"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7","Type":"ContainerStarted","Data":"f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b"}
Nov 03 20:22:36.268050 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:36.267674    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g" event={"ID":"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7","Type":"ContainerStarted","Data":"7e2a308f8de53b1d1eceae704f3c4a26a0237c44d014a59bbec9e91da9350c79"}
Nov 03 20:22:36.314457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:36.314351    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g" podStartSLOduration=2.314329351 podStartE2EDuration="2.314329351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:36.301904445 +0000 UTC m=+2538.814726228" watchObservedRunningTime="2026-11-03 20:22:36.314329351 +0000 UTC m=+2538.827151135"
Nov 03 20:22:37.018494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:37.017988    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:37.022047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:37.021956    5600 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:22:37.022500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:37.022467    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g" podUID="2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 20:22:38.021457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:38.020800    5600 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:22:38.021457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:38.020873    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g" podUID="2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 20:22:39.016829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:39.016792    5600 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:22:39.017124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:39.017090    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g" podUID="2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 20:22:40.025456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:40.017953    5600 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:22:40.025456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:40.018019    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g" podUID="2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 20:22:41.016640 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:41.016596    5600 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:22:41.016853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:41.016677    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g" podUID="2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 20:22:42.314465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:42.308799    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:42.314465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:42.310348    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:42.329761 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:42.329728    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:22:42.771911 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:42.771877    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:42.785218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:42.785188    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:43.413149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:43.413104    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:22:47.206369 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:22:47.206060    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"]
Nov 03 20:22:58.797347 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:22:58.797304    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:23:02.245160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.245102    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl" podUID="d92ae809-042d-4ddf-bd1d-f582025bea07" containerName="main" containerID="cri-o://15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126" gracePeriod=45
Nov 03 20:23:02.250291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.250252    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl" podUID="d92ae809-042d-4ddf-bd1d-f582025bea07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": EOF"
Nov 03 20:23:02.562927 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.562901    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:23:02.603741 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.603703    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-home\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-home\") pod \"d92ae809-042d-4ddf-bd1d-f582025bea07\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") "
Nov 03 20:23:02.603943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.603823    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d92ae809-042d-4ddf-bd1d-f582025bea07-tls-certs\" (UniqueName: \"kubernetes.io/secret/d92ae809-042d-4ddf-bd1d-f582025bea07-tls-certs\") pod \"d92ae809-042d-4ddf-bd1d-f582025bea07\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") "
Nov 03 20:23:02.603943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.603914    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92ae809-042d-4ddf-bd1d-f582025bea07-kube-api-access-p2pkt\" (UniqueName: \"kubernetes.io/projected/d92ae809-042d-4ddf-bd1d-f582025bea07-kube-api-access-p2pkt\") pod \"d92ae809-042d-4ddf-bd1d-f582025bea07\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") "
Nov 03 20:23:02.604069 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.604018    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-dshm\") pod \"d92ae809-042d-4ddf-bd1d-f582025bea07\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") "
Nov 03 20:23:02.604069 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.604045    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-model-cache\") pod \"d92ae809-042d-4ddf-bd1d-f582025bea07\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") "
Nov 03 20:23:02.604186 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.604087    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-tmp-dir\") pod \"d92ae809-042d-4ddf-bd1d-f582025bea07\" (UID: \"d92ae809-042d-4ddf-bd1d-f582025bea07\") "
Nov 03 20:23:02.605138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.605109    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-home" pod "d92ae809-042d-4ddf-bd1d-f582025bea07" (UID: "d92ae809-042d-4ddf-bd1d-f582025bea07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:02.610677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.610537    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d92ae809-042d-4ddf-bd1d-f582025bea07-tls-certs" pod "d92ae809-042d-4ddf-bd1d-f582025bea07" (UID: "d92ae809-042d-4ddf-bd1d-f582025bea07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:02.611688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.611652    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-tmp-dir" pod "d92ae809-042d-4ddf-bd1d-f582025bea07" (UID: "d92ae809-042d-4ddf-bd1d-f582025bea07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:02.612132 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.612108    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-model-cache" pod "d92ae809-042d-4ddf-bd1d-f582025bea07" (UID: "d92ae809-042d-4ddf-bd1d-f582025bea07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:02.622180 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.622132    5600 generic.go:356] "Generic (PLEG): container finished" podID="d92ae809-042d-4ddf-bd1d-f582025bea07" containerID="15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126" exitCode=0
Nov 03 20:23:02.622417 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.622189    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"
Nov 03 20:23:02.622621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.622211    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl" event={"ID":"d92ae809-042d-4ddf-bd1d-f582025bea07","Type":"ContainerDied","Data":"15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126"}
Nov 03 20:23:02.622891 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.622724    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl" event={"ID":"d92ae809-042d-4ddf-bd1d-f582025bea07","Type":"ContainerDied","Data":"5ced858db382bb6941aa3b890ae31820ce913f86b28db4968afcf3c87592a0c8"}
Nov 03 20:23:02.623036 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.622846    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-dshm" pod "d92ae809-042d-4ddf-bd1d-f582025bea07" (UID: "d92ae809-042d-4ddf-bd1d-f582025bea07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:02.623036 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.622870    5600 scope.go:122] "RemoveContainer" containerID="15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126"
Nov 03 20:23:02.624763 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.624739    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92ae809-042d-4ddf-bd1d-f582025bea07-kube-api-access-p2pkt" pod "d92ae809-042d-4ddf-bd1d-f582025bea07" (UID: "d92ae809-042d-4ddf-bd1d-f582025bea07"). InnerVolumeSpecName "kube-api-access-p2pkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:02.657712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.657687    5600 scope.go:122] "RemoveContainer" containerID="15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126"
Nov 03 20:23:02.658091 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:23:02.658057    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126\": container with ID starting with 15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126 not found: ID does not exist" containerID="15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126"
Nov 03 20:23:02.658149 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.658104    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126"} err="failed to get container status \"15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126\": rpc error: code = NotFound desc = could not find container \"15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126\": container with ID starting with 15967f978c6d07dd40ebe92152478f54f4256a0811ced0326157ef1af5ed5126 not found: ID does not exist"
Nov 03 20:23:02.706398 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.706370    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:02.706398 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.706400    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92ae809-042d-4ddf-bd1d-f582025bea07-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:02.706642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.706412    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2pkt\" (UniqueName: \"kubernetes.io/projected/d92ae809-042d-4ddf-bd1d-f582025bea07-kube-api-access-p2pkt\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:02.706642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.706422    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:02.706642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.706450    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:02.706642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.706464    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92ae809-042d-4ddf-bd1d-f582025bea07-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:02.952490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.951814    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"]
Nov 03 20:23:02.956581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:02.956552    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a8e19e/auth-test-model-a8e19e-kserve-66646678bf-jlqxl"]
Nov 03 20:23:03.621542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:03.621512    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92ae809-042d-4ddf-bd1d-f582025bea07" path="/var/lib/kubelet/pods/d92ae809-042d-4ddf-bd1d-f582025bea07/volumes"
Nov 03 20:23:04.731553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:04.731519    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"]
Nov 03 20:23:04.732324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:04.732284    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt" podUID="8985a09b-1f37-4792-9588-389ecdafe353" containerName="maas-api" containerID="cri-o://bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94" gracePeriod=30
Nov 03 20:23:05.325488 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.321875    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:23:05.424377 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.424344    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-metrics-certs\") pod \"8985a09b-1f37-4792-9588-389ecdafe353\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") "
Nov 03 20:23:05.424592 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.424397    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8985a09b-1f37-4792-9588-389ecdafe353-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8985a09b-1f37-4792-9588-389ecdafe353-openshift-service-ca\") pod \"8985a09b-1f37-4792-9588-389ecdafe353\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") "
Nov 03 20:23:05.424592 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.424527    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-maas-api-tls\") pod \"8985a09b-1f37-4792-9588-389ecdafe353\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") "
Nov 03 20:23:05.424716 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.424657    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8985a09b-1f37-4792-9588-389ecdafe353-kube-api-access-fhqmx\" (UniqueName: \"kubernetes.io/projected/8985a09b-1f37-4792-9588-389ecdafe353-kube-api-access-fhqmx\") pod \"8985a09b-1f37-4792-9588-389ecdafe353\" (UID: \"8985a09b-1f37-4792-9588-389ecdafe353\") "
Nov 03 20:23:05.425321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.425291    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8985a09b-1f37-4792-9588-389ecdafe353-openshift-service-ca" pod "8985a09b-1f37-4792-9588-389ecdafe353" (UID: "8985a09b-1f37-4792-9588-389ecdafe353"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:05.425596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.425579    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8985a09b-1f37-4792-9588-389ecdafe353-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:05.427766 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.427732    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-metrics-certs" pod "8985a09b-1f37-4792-9588-389ecdafe353" (UID: "8985a09b-1f37-4792-9588-389ecdafe353"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:05.427766 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.427743    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8985a09b-1f37-4792-9588-389ecdafe353-kube-api-access-fhqmx" pod "8985a09b-1f37-4792-9588-389ecdafe353" (UID: "8985a09b-1f37-4792-9588-389ecdafe353"). InnerVolumeSpecName "kube-api-access-fhqmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:05.430767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.430740    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-maas-api-tls" pod "8985a09b-1f37-4792-9588-389ecdafe353" (UID: "8985a09b-1f37-4792-9588-389ecdafe353"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:05.527278 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.527191    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:05.527278 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.527230    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8985a09b-1f37-4792-9588-389ecdafe353-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:05.527278 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.527245    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhqmx\" (UniqueName: \"kubernetes.io/projected/8985a09b-1f37-4792-9588-389ecdafe353-kube-api-access-fhqmx\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:05.650336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.650297    5600 generic.go:356] "Generic (PLEG): container finished" podID="8985a09b-1f37-4792-9588-389ecdafe353" containerID="bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94" exitCode=0
Nov 03 20:23:05.650525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.650352    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"
Nov 03 20:23:05.650525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.650354    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt" event={"ID":"8985a09b-1f37-4792-9588-389ecdafe353","Type":"ContainerDied","Data":"bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94"}
Nov 03 20:23:05.650525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.650407    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt" event={"ID":"8985a09b-1f37-4792-9588-389ecdafe353","Type":"ContainerDied","Data":"a7d3f0a3058aaf019db8b073b45a4342236bb5e92d29c900aff5786d963080ad"}
Nov 03 20:23:05.650525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.650445    5600 scope.go:122] "RemoveContainer" containerID="bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94"
Nov 03 20:23:05.662026 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.661997    5600 scope.go:122] "RemoveContainer" containerID="bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94"
Nov 03 20:23:05.662312 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:23:05.662294    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94\": container with ID starting with bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94 not found: ID does not exist" containerID="bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94"
Nov 03 20:23:05.662387 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.662325    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94"} err="failed to get container status \"bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94\": rpc error: code = NotFound desc = could not find container \"bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94\": container with ID starting with bf8b3f6c64b5c25efe6e77d65b7ae1ae297cc095ed35a5778b7ff416b63a8b94 not found: ID does not exist"
Nov 03 20:23:05.671556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.671534    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"]
Nov 03 20:23:05.675709 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:05.675690    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-9ff9f5-5759546db4-vsrlt"]
Nov 03 20:23:07.620169 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:07.620130    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8985a09b-1f37-4792-9588-389ecdafe353" path="/var/lib/kubelet/pods/8985a09b-1f37-4792-9588-389ecdafe353/volumes"
Nov 03 20:23:28.814840 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:23:28.814797    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:23:31.126298 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.126262    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"]
Nov 03 20:23:31.127473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.127410    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l" podUID="011eb7d6-11dc-4fa6-8bfa-82897107b779" containerName="maas-api" containerID="cri-o://a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225" gracePeriod=30
Nov 03 20:23:31.777211 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.776852    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:23:31.948327 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.948295    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-maas-api-tls\") pod \"011eb7d6-11dc-4fa6-8bfa-82897107b779\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") "
Nov 03 20:23:31.948585 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.948348    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/011eb7d6-11dc-4fa6-8bfa-82897107b779-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/011eb7d6-11dc-4fa6-8bfa-82897107b779-openshift-service-ca\") pod \"011eb7d6-11dc-4fa6-8bfa-82897107b779\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") "
Nov 03 20:23:31.948585 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.948452    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-metrics-certs\" (UniqueName: \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-metrics-certs\") pod \"011eb7d6-11dc-4fa6-8bfa-82897107b779\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") "
Nov 03 20:23:31.948585 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.948473    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011eb7d6-11dc-4fa6-8bfa-82897107b779-kube-api-access-n4njs\" (UniqueName: \"kubernetes.io/projected/011eb7d6-11dc-4fa6-8bfa-82897107b779-kube-api-access-n4njs\") pod \"011eb7d6-11dc-4fa6-8bfa-82897107b779\" (UID: \"011eb7d6-11dc-4fa6-8bfa-82897107b779\") "
Nov 03 20:23:31.948944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.948920    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011eb7d6-11dc-4fa6-8bfa-82897107b779-openshift-service-ca" pod "011eb7d6-11dc-4fa6-8bfa-82897107b779" (UID: "011eb7d6-11dc-4fa6-8bfa-82897107b779"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:31.949089 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.949065    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/011eb7d6-11dc-4fa6-8bfa-82897107b779-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:31.951141 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.951110    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-maas-api-tls" pod "011eb7d6-11dc-4fa6-8bfa-82897107b779" (UID: "011eb7d6-11dc-4fa6-8bfa-82897107b779"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:31.951256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.951176    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-metrics-certs" pod "011eb7d6-11dc-4fa6-8bfa-82897107b779" (UID: "011eb7d6-11dc-4fa6-8bfa-82897107b779"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:31.951256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:31.951234    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011eb7d6-11dc-4fa6-8bfa-82897107b779-kube-api-access-n4njs" pod "011eb7d6-11dc-4fa6-8bfa-82897107b779" (UID: "011eb7d6-11dc-4fa6-8bfa-82897107b779"). InnerVolumeSpecName "kube-api-access-n4njs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:32.031687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.031590    5600 generic.go:356] "Generic (PLEG): container finished" podID="011eb7d6-11dc-4fa6-8bfa-82897107b779" containerID="a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225" exitCode=0
Nov 03 20:23:32.031687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.031639    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"
Nov 03 20:23:32.031687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.031668    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l" event={"ID":"011eb7d6-11dc-4fa6-8bfa-82897107b779","Type":"ContainerDied","Data":"a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225"}
Nov 03 20:23:32.031966 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.031732    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l" event={"ID":"011eb7d6-11dc-4fa6-8bfa-82897107b779","Type":"ContainerDied","Data":"8e521cc1727be8a3b3f3ac25bf0b374acfdd61b95c414db10c18bdc206c704bc"}
Nov 03 20:23:32.031966 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.031761    5600 scope.go:122] "RemoveContainer" containerID="a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225"
Nov 03 20:23:32.043642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.043621    5600 scope.go:122] "RemoveContainer" containerID="a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225"
Nov 03 20:23:32.043949 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:23:32.043920    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225\": container with ID starting with a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225 not found: ID does not exist" containerID="a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225"
Nov 03 20:23:32.044017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.043961    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225"} err="failed to get container status \"a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225\": rpc error: code = NotFound desc = could not find container \"a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225\": container with ID starting with a1c0bb3f87aadbf6867a03c40a47bf25fae41d97719a127f02f14d1d5e71f225 not found: ID does not exist"
Nov 03 20:23:32.050805 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.050783    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:32.050805 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.050806    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011eb7d6-11dc-4fa6-8bfa-82897107b779-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:32.050939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.050818    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4njs\" (UniqueName: \"kubernetes.io/projected/011eb7d6-11dc-4fa6-8bfa-82897107b779-kube-api-access-n4njs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:32.056561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.056537    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"]
Nov 03 20:23:32.060185 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:32.060166    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a8e19e-7dc56fc467-nz72l"]
Nov 03 20:23:33.619518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:33.619480    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="011eb7d6-11dc-4fa6-8bfa-82897107b779" path="/var/lib/kubelet/pods/011eb7d6-11dc-4fa6-8bfa-82897107b779/volumes"
Nov 03 20:23:34.589113 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.589074    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"]
Nov 03 20:23:34.589900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.589873    5600 cpu_manager.go:414] "containerMap: removing container" podUID="d92ae809-042d-4ddf-bd1d-f582025bea07" containerName="main"
Nov 03 20:23:34.589900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.589898    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92ae809-042d-4ddf-bd1d-f582025bea07" containerName="main"
Nov 03 20:23:34.590116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.589948    5600 cpu_manager.go:414] "containerMap: removing container" podUID="011eb7d6-11dc-4fa6-8bfa-82897107b779" containerName="maas-api"
Nov 03 20:23:34.590116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.589955    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011eb7d6-11dc-4fa6-8bfa-82897107b779" containerName="maas-api"
Nov 03 20:23:34.590116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.589979    5600 cpu_manager.go:414] "containerMap: removing container" podUID="8985a09b-1f37-4792-9588-389ecdafe353" containerName="maas-api"
Nov 03 20:23:34.590116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.589985    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8985a09b-1f37-4792-9588-389ecdafe353" containerName="maas-api"
Nov 03 20:23:34.590116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.590093    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92ae809-042d-4ddf-bd1d-f582025bea07" containerName="main"
Nov 03 20:23:34.590116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.590105    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="011eb7d6-11dc-4fa6-8bfa-82897107b779" containerName="maas-api"
Nov 03 20:23:34.590116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.590115    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="8985a09b-1f37-4792-9588-389ecdafe353" containerName="maas-api"
Nov 03 20:23:34.595754 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.595726    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.622894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.622863    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"]
Nov 03 20:23:34.697726 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.697687    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.697919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.697738    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.697919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.697871    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.698040 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.697999    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.698093 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.698071    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrphg\" (UniqueName: \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-kube-api-access-mrphg\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.698192 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.698172    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.698242 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.698209    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.698400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.698376    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.698537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.698516    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.800370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.800334    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.800609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.800381    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.800609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.800540    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.800609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.800601    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.800780 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.800675    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.800780 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.800700    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.800780 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.800742    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.800914 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.800897    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.801521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.801497    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.802040 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.802011    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.802166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.802085    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrphg\" (UniqueName: \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-kube-api-access-mrphg\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.802702 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.802666    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.802856 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.802808    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.803420 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.803367    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.804349 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.804329    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.811711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.811683    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.813302 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.813276    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrphg\" (UniqueName: \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-kube-api-access-mrphg\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.816793 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.816771    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:34.924252 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:34.924217    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:35.216643 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:35.216612    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"]
Nov 03 20:23:35.217969 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:23:35.217943    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0b36e6_38af_4c60_a4a1_3f3c0ee8b338.slice/crio-5d46d029856c37b339eef0ebea09370c4dc357c8e741dd72fdbeddefacc881ca WatchSource:0}: Error finding container 5d46d029856c37b339eef0ebea09370c4dc357c8e741dd72fdbeddefacc881ca: Status 404 returned error can't find the container with id 5d46d029856c37b339eef0ebea09370c4dc357c8e741dd72fdbeddefacc881ca
Nov 03 20:23:35.221269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:35.221235    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:23:35.221393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:35.221318    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:23:35.221393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:35.221364    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:23:36.080288 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:36.080247    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" event={"ID":"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338","Type":"ContainerStarted","Data":"fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410"}
Nov 03 20:23:36.080846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:36.080315    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" event={"ID":"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338","Type":"ContainerStarted","Data":"5d46d029856c37b339eef0ebea09370c4dc357c8e741dd72fdbeddefacc881ca"}
Nov 03 20:23:36.150482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:36.143561    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" podStartSLOduration=2.143535957 podStartE2EDuration="2.143535957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:36.117954448 +0000 UTC m=+2598.630776231" watchObservedRunningTime="2026-11-03 20:23:36.143535957 +0000 UTC m=+2598.656357739"
Nov 03 20:23:36.927469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:36.924807    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:36.931467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:36.931138    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:23:36.931467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:36.931230    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" podUID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:23:37.925627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:37.925586    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:23:37.926403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:37.926363    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" podUID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:23:38.925179 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:38.925146    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:23:38.926386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:38.925198    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" podUID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:23:39.935837 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:39.935782    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:23:39.936492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:39.936456    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" podUID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:23:40.925394 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:40.925358    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:23:40.926212 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:40.925416    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" podUID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:23:42.021832 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:42.021800    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:42.023117 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:42.023085    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:42.024305 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:42.024281    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:23:43.631547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:43.631509    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"]
Nov 03 20:23:58.659649 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:58.659554    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main" containerID="cri-o://ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7" gracePeriod=45
Nov 03 20:23:58.834733 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:23:58.830804    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:23:58.983891 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:58.983864    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:23:59.067198 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.067126    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c544766f-9820-441d-9564-647d963fa1ea-kube-api-access-2pfnn\" (UniqueName: \"kubernetes.io/projected/c544766f-9820-441d-9564-647d963fa1ea-kube-api-access-2pfnn\") pod \"c544766f-9820-441d-9564-647d963fa1ea\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") "
Nov 03 20:23:59.067458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.067234    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c544766f-9820-441d-9564-647d963fa1ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/c544766f-9820-441d-9564-647d963fa1ea-tls-certs\") pod \"c544766f-9820-441d-9564-647d963fa1ea\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") "
Nov 03 20:23:59.067458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.067274    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-home\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-home\") pod \"c544766f-9820-441d-9564-647d963fa1ea\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") "
Nov 03 20:23:59.067458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.067317    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-dshm\") pod \"c544766f-9820-441d-9564-647d963fa1ea\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") "
Nov 03 20:23:59.067458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.067411    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-tmp-dir\") pod \"c544766f-9820-441d-9564-647d963fa1ea\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") "
Nov 03 20:23:59.067816 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.067472    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-model-cache\") pod \"c544766f-9820-441d-9564-647d963fa1ea\" (UID: \"c544766f-9820-441d-9564-647d963fa1ea\") "
Nov 03 20:23:59.068571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.068502    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-home" pod "c544766f-9820-441d-9564-647d963fa1ea" (UID: "c544766f-9820-441d-9564-647d963fa1ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.068689 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.068571    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-model-cache" pod "c544766f-9820-441d-9564-647d963fa1ea" (UID: "c544766f-9820-441d-9564-647d963fa1ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.068767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.068693    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-tmp-dir" pod "c544766f-9820-441d-9564-647d963fa1ea" (UID: "c544766f-9820-441d-9564-647d963fa1ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.072152 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.072113    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c544766f-9820-441d-9564-647d963fa1ea-tls-certs" pod "c544766f-9820-441d-9564-647d963fa1ea" (UID: "c544766f-9820-441d-9564-647d963fa1ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:59.072409 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.072388    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c544766f-9820-441d-9564-647d963fa1ea-kube-api-access-2pfnn" pod "c544766f-9820-441d-9564-647d963fa1ea" (UID: "c544766f-9820-441d-9564-647d963fa1ea"). InnerVolumeSpecName "kube-api-access-2pfnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:59.072546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.072508    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-dshm" pod "c544766f-9820-441d-9564-647d963fa1ea" (UID: "c544766f-9820-441d-9564-647d963fa1ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.169569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.169536    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pfnn\" (UniqueName: \"kubernetes.io/projected/c544766f-9820-441d-9564-647d963fa1ea-kube-api-access-2pfnn\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.169569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.169565    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c544766f-9820-441d-9564-647d963fa1ea-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.169569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.169575    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.169800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.169583    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.169800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.169590    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.169800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.169599    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c544766f-9820-441d-9564-647d963fa1ea-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.329336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.329238    5600 generic.go:356] "Generic (PLEG): container finished" podID="c544766f-9820-441d-9564-647d963fa1ea" containerID="ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7" exitCode=0
Nov 03 20:23:59.329336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.329296    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"
Nov 03 20:23:59.329336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.329316    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" event={"ID":"c544766f-9820-441d-9564-647d963fa1ea","Type":"ContainerDied","Data":"ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7"}
Nov 03 20:23:59.329683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.329365    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s" event={"ID":"c544766f-9820-441d-9564-647d963fa1ea","Type":"ContainerDied","Data":"ca880659d5900c1c16f3c80140cff3ddee5bec7bdcdd73925170db68d18d5e26"}
Nov 03 20:23:59.329683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.329382    5600 scope.go:122] "RemoveContainer" containerID="ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7"
Nov 03 20:23:59.341334 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.341317    5600 scope.go:122] "RemoveContainer" containerID="ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7"
Nov 03 20:23:59.341964 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:23:59.341938    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7\": container with ID starting with ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7 not found: ID does not exist" containerID="ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7"
Nov 03 20:23:59.342037 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.341972    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7"} err="failed to get container status \"ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7\": rpc error: code = NotFound desc = could not find container \"ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7\": container with ID starting with ccd5662b80da0dc87a87966398f9bbe1dd90d3abb19fd0c6c30dd9739fba7aa7 not found: ID does not exist"
Nov 03 20:23:59.354846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.354821    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"]
Nov 03 20:23:59.358663 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.358641    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/simulator-w3-b87733-kserve-65b6f56976-b6l7s"]
Nov 03 20:23:59.622156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:23:59.622060    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c544766f-9820-441d-9564-647d963fa1ea" path="/var/lib/kubelet/pods/c544766f-9820-441d-9564-647d963fa1ea/volumes"
Nov 03 20:24:11.152668 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.151520    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"]
Nov 03 20:24:11.153335 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.153287    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph" podUID="176d8e7c-0897-4a2a-bb67-cda280625d1e" containerName="maas-api" containerID="cri-o://ff76487e37c2631acc1a93cc5bf03c93372d531323058f3eb601437b2fe44736" gracePeriod=30
Nov 03 20:24:11.703787 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.703595    5600 generic.go:356] "Generic (PLEG): container finished" podID="176d8e7c-0897-4a2a-bb67-cda280625d1e" containerID="ff76487e37c2631acc1a93cc5bf03c93372d531323058f3eb601437b2fe44736" exitCode=0
Nov 03 20:24:11.761379 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.760174    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph" event={"ID":"176d8e7c-0897-4a2a-bb67-cda280625d1e","Type":"ContainerDied","Data":"ff76487e37c2631acc1a93cc5bf03c93372d531323058f3eb601437b2fe44736"}
Nov 03 20:24:11.865119 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.865086    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:24:11.910391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.909999    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-metrics-certs\") pod \"176d8e7c-0897-4a2a-bb67-cda280625d1e\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") "
Nov 03 20:24:11.910391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.910111    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176d8e7c-0897-4a2a-bb67-cda280625d1e-kube-api-access-z2zz4\" (UniqueName: \"kubernetes.io/projected/176d8e7c-0897-4a2a-bb67-cda280625d1e-kube-api-access-z2zz4\") pod \"176d8e7c-0897-4a2a-bb67-cda280625d1e\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") "
Nov 03 20:24:11.910391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.910208    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176d8e7c-0897-4a2a-bb67-cda280625d1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176d8e7c-0897-4a2a-bb67-cda280625d1e-openshift-service-ca\") pod \"176d8e7c-0897-4a2a-bb67-cda280625d1e\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") "
Nov 03 20:24:11.910391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.910285    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-maas-api-tls\") pod \"176d8e7c-0897-4a2a-bb67-cda280625d1e\" (UID: \"176d8e7c-0897-4a2a-bb67-cda280625d1e\") "
Nov 03 20:24:11.910785 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.910684    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176d8e7c-0897-4a2a-bb67-cda280625d1e-openshift-service-ca" pod "176d8e7c-0897-4a2a-bb67-cda280625d1e" (UID: "176d8e7c-0897-4a2a-bb67-cda280625d1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:11.911162 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.911135    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176d8e7c-0897-4a2a-bb67-cda280625d1e-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:11.916657 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.915004    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176d8e7c-0897-4a2a-bb67-cda280625d1e-kube-api-access-z2zz4" pod "176d8e7c-0897-4a2a-bb67-cda280625d1e" (UID: "176d8e7c-0897-4a2a-bb67-cda280625d1e"). InnerVolumeSpecName "kube-api-access-z2zz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:11.917378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.917358    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-metrics-certs" pod "176d8e7c-0897-4a2a-bb67-cda280625d1e" (UID: "176d8e7c-0897-4a2a-bb67-cda280625d1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:11.917561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:11.917418    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-maas-api-tls" pod "176d8e7c-0897-4a2a-bb67-cda280625d1e" (UID: "176d8e7c-0897-4a2a-bb67-cda280625d1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:12.017609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:12.017518    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:12.017609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:12.017553    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2zz4\" (UniqueName: \"kubernetes.io/projected/176d8e7c-0897-4a2a-bb67-cda280625d1e-kube-api-access-z2zz4\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:12.017609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:12.017570    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176d8e7c-0897-4a2a-bb67-cda280625d1e-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:12.721922 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:12.721892    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"
Nov 03 20:24:12.722468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:12.721927    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph" event={"ID":"176d8e7c-0897-4a2a-bb67-cda280625d1e","Type":"ContainerDied","Data":"a0d98b33bec2e8c257260cc0233c46a5be574fbc3a941f2e269835db8087e6f7"}
Nov 03 20:24:12.722468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:12.721997    5600 scope.go:122] "RemoveContainer" containerID="ff76487e37c2631acc1a93cc5bf03c93372d531323058f3eb601437b2fe44736"
Nov 03 20:24:12.760112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:12.760085    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"]
Nov 03 20:24:12.763751 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:12.763729    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b87733-6b584b887b-sq7ph"]
Nov 03 20:24:13.620372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:13.620339    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176d8e7c-0897-4a2a-bb67-cda280625d1e" path="/var/lib/kubelet/pods/176d8e7c-0897-4a2a-bb67-cda280625d1e/volumes"
Nov 03 20:24:19.332453 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:19.312983    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"]
Nov 03 20:24:19.332453 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:19.313224    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy" containerID="cri-o://5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211" gracePeriod=30
Nov 03 20:24:24.682704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.682670    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:24:24.793650 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.793609    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-podinfo\") pod \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") "
Nov 03 20:24:24.793862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.793698    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istiod-ca-cert\") pod \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") "
Nov 03 20:24:24.793862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.793749    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-data\") pod \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") "
Nov 03 20:24:24.793862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.793792    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-envoy\") pod \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") "
Nov 03 20:24:24.794033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.793861    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-certs\") pod \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") "
Nov 03 20:24:24.794033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.793988    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-socket\") pod \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") "
Nov 03 20:24:24.794140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.794072    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-token\" (UniqueName: \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-token\") pod \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") "
Nov 03 20:24:24.794487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.794456    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-certs" pod "71d7ec61-4c37-4cfc-80fe-102b094dfe0c" (UID: "71d7ec61-4c37-4cfc-80fe-102b094dfe0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:24.794604 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.794577    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istiod-ca-cert" pod "71d7ec61-4c37-4cfc-80fe-102b094dfe0c" (UID: "71d7ec61-4c37-4cfc-80fe-102b094dfe0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:24.794727 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.794708    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-kube-api-access-hzcjl\" (UniqueName: \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-kube-api-access-hzcjl\") pod \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") "
Nov 03 20:24:24.794860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.794840    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-credential-socket\") pod \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\" (UID: \"71d7ec61-4c37-4cfc-80fe-102b094dfe0c\") "
Nov 03 20:24:24.794918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.794880    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-socket" pod "71d7ec61-4c37-4cfc-80fe-102b094dfe0c" (UID: "71d7ec61-4c37-4cfc-80fe-102b094dfe0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:24.795347 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.795320    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-data" pod "71d7ec61-4c37-4cfc-80fe-102b094dfe0c" (UID: "71d7ec61-4c37-4cfc-80fe-102b094dfe0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:24.795634 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.795608    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-credential-socket" pod "71d7ec61-4c37-4cfc-80fe-102b094dfe0c" (UID: "71d7ec61-4c37-4cfc-80fe-102b094dfe0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:24.795799 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.795777    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.795858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.795806    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.795858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.795824    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.795858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.795838    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.795858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.795852    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.797565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.797536    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-envoy" pod "71d7ec61-4c37-4cfc-80fe-102b094dfe0c" (UID: "71d7ec61-4c37-4cfc-80fe-102b094dfe0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:24.797678 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.797606    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-podinfo" pod "71d7ec61-4c37-4cfc-80fe-102b094dfe0c" (UID: "71d7ec61-4c37-4cfc-80fe-102b094dfe0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:24:24.798156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.798129    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-token" pod "71d7ec61-4c37-4cfc-80fe-102b094dfe0c" (UID: "71d7ec61-4c37-4cfc-80fe-102b094dfe0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:24.798750 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.798729    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-kube-api-access-hzcjl" pod "71d7ec61-4c37-4cfc-80fe-102b094dfe0c" (UID: "71d7ec61-4c37-4cfc-80fe-102b094dfe0c"). InnerVolumeSpecName "kube-api-access-hzcjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:24.840690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.840651    5600 generic.go:356] "Generic (PLEG): container finished" podID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerID="5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211" exitCode=0
Nov 03 20:24:24.840690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.840699    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"
Nov 03 20:24:24.840929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.840733    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" event={"ID":"71d7ec61-4c37-4cfc-80fe-102b094dfe0c","Type":"ContainerDied","Data":"5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211"}
Nov 03 20:24:24.840929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.840793    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s" event={"ID":"71d7ec61-4c37-4cfc-80fe-102b094dfe0c","Type":"ContainerDied","Data":"80afbeaa772d689161c0878e8c8c45a35e3247914629c94dd3499ee7e5df07d3"}
Nov 03 20:24:24.840929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.840823    5600 scope.go:122] "RemoveContainer" containerID="5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211"
Nov 03 20:24:24.854105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.854081    5600 scope.go:122] "RemoveContainer" containerID="5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211"
Nov 03 20:24:24.854588 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:24:24.854562    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211\": container with ID starting with 5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211 not found: ID does not exist" containerID="5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211"
Nov 03 20:24:24.854680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.854602    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211"} err="failed to get container status \"5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211\": rpc error: code = NotFound desc = could not find container \"5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211\": container with ID starting with 5fabfa2c84314cac0244c70fc9442aaf3c20d5866e594ae0fe705b9b0f4bb211 not found: ID does not exist"
Nov 03 20:24:24.872503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.872362    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"]
Nov 03 20:24:24.876813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.876786    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-5zp2s"]
Nov 03 20:24:24.897159 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.897131    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzcjl\" (UniqueName: \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-kube-api-access-hzcjl\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.897159 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.897161    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.897380 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.897175    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.897380 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:24.897189    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d7ec61-4c37-4cfc-80fe-102b094dfe0c-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:25.642769 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:24:25.640904    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" path="/var/lib/kubelet/pods/71d7ec61-4c37-4cfc-80fe-102b094dfe0c/volumes"
Nov 03 20:24:28.805845 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:24:28.805804    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:24:58.811594 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:24:58.811553    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:25:25.588108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:25.588065    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"]
Nov 03 20:25:28.805147 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:25:28.805050    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:25:29.054102 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.054062    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"]
Nov 03 20:25:29.055255 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.054965    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2" podUID="a243ff65-42fa-4c4c-97a0-4ed640695dee" containerName="maas-api" containerID="cri-o://67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b" gracePeriod=30
Nov 03 20:25:29.201626 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.201554    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"]
Nov 03 20:25:29.824915 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.824884    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:25:29.909552 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.909514    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a243ff65-42fa-4c4c-97a0-4ed640695dee-kube-api-access-qz2pd\" (UniqueName: \"kubernetes.io/projected/a243ff65-42fa-4c4c-97a0-4ed640695dee-kube-api-access-qz2pd\") pod \"a243ff65-42fa-4c4c-97a0-4ed640695dee\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") "
Nov 03 20:25:29.909770 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.909607    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-maas-api-tls\") pod \"a243ff65-42fa-4c4c-97a0-4ed640695dee\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") "
Nov 03 20:25:29.909882 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.909860    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a243ff65-42fa-4c4c-97a0-4ed640695dee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a243ff65-42fa-4c4c-97a0-4ed640695dee-openshift-service-ca\") pod \"a243ff65-42fa-4c4c-97a0-4ed640695dee\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") "
Nov 03 20:25:29.910215 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.910195    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-metrics-certs\") pod \"a243ff65-42fa-4c4c-97a0-4ed640695dee\" (UID: \"a243ff65-42fa-4c4c-97a0-4ed640695dee\") "
Nov 03 20:25:29.910404 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.910191    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a243ff65-42fa-4c4c-97a0-4ed640695dee-openshift-service-ca" pod "a243ff65-42fa-4c4c-97a0-4ed640695dee" (UID: "a243ff65-42fa-4c4c-97a0-4ed640695dee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:29.911202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.911174    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a243ff65-42fa-4c4c-97a0-4ed640695dee-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:29.912510 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.912420    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-metrics-certs" pod "a243ff65-42fa-4c4c-97a0-4ed640695dee" (UID: "a243ff65-42fa-4c4c-97a0-4ed640695dee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:29.912615 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.912541    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-maas-api-tls" pod "a243ff65-42fa-4c4c-97a0-4ed640695dee" (UID: "a243ff65-42fa-4c4c-97a0-4ed640695dee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:29.912615 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:29.912587    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a243ff65-42fa-4c4c-97a0-4ed640695dee-kube-api-access-qz2pd" pod "a243ff65-42fa-4c4c-97a0-4ed640695dee" (UID: "a243ff65-42fa-4c4c-97a0-4ed640695dee"). InnerVolumeSpecName "kube-api-access-qz2pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:30.013990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.013089    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:30.013990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.013130    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz2pd\" (UniqueName: \"kubernetes.io/projected/a243ff65-42fa-4c4c-97a0-4ed640695dee-kube-api-access-qz2pd\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:30.013990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.013146    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a243ff65-42fa-4c4c-97a0-4ed640695dee-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:30.527164 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.527129    5600 generic.go:356] "Generic (PLEG): container finished" podID="a243ff65-42fa-4c4c-97a0-4ed640695dee" containerID="67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b" exitCode=0
Nov 03 20:25:30.527391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.527198    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"
Nov 03 20:25:30.527391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.527191    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2" event={"ID":"a243ff65-42fa-4c4c-97a0-4ed640695dee","Type":"ContainerDied","Data":"67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b"}
Nov 03 20:25:30.527391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.527306    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2" event={"ID":"a243ff65-42fa-4c4c-97a0-4ed640695dee","Type":"ContainerDied","Data":"c28628a91ceacc6487d40102f989dd08062446ccafa5935c1b109469acd0a100"}
Nov 03 20:25:30.527391 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.527356    5600 scope.go:122] "RemoveContainer" containerID="67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b"
Nov 03 20:25:30.541405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.541379    5600 scope.go:122] "RemoveContainer" containerID="67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b"
Nov 03 20:25:30.541846 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:25:30.541812    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b\": container with ID starting with 67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b not found: ID does not exist" containerID="67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b"
Nov 03 20:25:30.541929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.541854    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b"} err="failed to get container status \"67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b\": rpc error: code = NotFound desc = could not find container \"67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b\": container with ID starting with 67831d13c0de7aa83199082fcaeb0c400ce36e92a855c5aa8c36083f3ec5626b not found: ID does not exist"
Nov 03 20:25:30.553181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.553153    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"]
Nov 03 20:25:30.556632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.556607    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-46c34c-7cd7b46545-hkhb2"]
Nov 03 20:25:30.589324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:30.589275    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7" podUID="441b3899-e37a-4c0a-90e4-8cbf1ece2c65" containerName="payload-pre-processing" containerID="cri-o://227c7f9079dc2c4381767925d75ea019f246220d1bf7596e03e3b802a2377637" gracePeriod=25
Nov 03 20:25:31.619762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:31.619723    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a243ff65-42fa-4c4c-97a0-4ed640695dee" path="/var/lib/kubelet/pods/a243ff65-42fa-4c4c-97a0-4ed640695dee/volumes"
Nov 03 20:25:32.912020 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:32.911750    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"]
Nov 03 20:25:36.246465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:36.244863    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"]
Nov 03 20:25:36.246465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:36.245104    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g" podUID="2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" containerName="istio-proxy" containerID="cri-o://f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b" gracePeriod=30
Nov 03 20:25:36.513513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:36.495391    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"]
Nov 03 20:25:36.786531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:36.783059    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7" podUID="441b3899-e37a-4c0a-90e4-8cbf1ece2c65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:25:41.015590 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.015548    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"]
Nov 03 20:25:41.636710 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.636672    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:25:41.723797 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.723743    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-token\" (UniqueName: \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-token\") pod \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") "
Nov 03 20:25:41.724833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.724722    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-credential-socket\") pod \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") "
Nov 03 20:25:41.725523 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.725341    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-credential-socket" pod "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" (UID: "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.726093 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.726069    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-socket" pod "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" (UID: "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.726279 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.725427    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-socket\") pod \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") "
Nov 03 20:25:41.726487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.726469    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-kube-api-access-gzkmj\" (UniqueName: \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-kube-api-access-gzkmj\") pod \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") "
Nov 03 20:25:41.727406 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.727043    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-data\") pod \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") "
Nov 03 20:25:41.727632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.727605    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istiod-ca-cert\") pod \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") "
Nov 03 20:25:41.728343 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.727372    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-data" pod "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" (UID: "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.728514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.728207    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istiod-ca-cert" pod "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" (UID: "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:41.729156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.729061    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-token" pod "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" (UID: "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:41.729372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.728307    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-envoy\") pod \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") "
Nov 03 20:25:41.729551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.729535    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-certs\") pod \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") "
Nov 03 20:25:41.730216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.730196    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-podinfo\") pod \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\" (UID: \"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7\") "
Nov 03 20:25:41.731708 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.730012    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-certs" pod "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" (UID: "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.732749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.732727    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.732884 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.732870    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.732986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.732972    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.733094 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.733081    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.733188 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.733176    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.733278 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.733267    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.733959 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.733933    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-podinfo" pod "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" (UID: "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:41.735325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.735308    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-envoy" pod "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" (UID: "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.735542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.735528    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-kube-api-access-gzkmj" pod "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" (UID: "2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7"). InnerVolumeSpecName "kube-api-access-gzkmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:41.750422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.750394    5600 generic.go:356] "Generic (PLEG): container finished" podID="2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" containerID="f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b" exitCode=0
Nov 03 20:25:41.750580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.750508    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"
Nov 03 20:25:41.750580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.750566    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g" event={"ID":"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7","Type":"ContainerDied","Data":"f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b"}
Nov 03 20:25:41.750702 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.750621    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g" event={"ID":"2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7","Type":"ContainerDied","Data":"7e2a308f8de53b1d1eceae704f3c4a26a0237c44d014a59bbec9e91da9350c79"}
Nov 03 20:25:41.750702 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.750653    5600 scope.go:122] "RemoveContainer" containerID="f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b"
Nov 03 20:25:41.763150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.763131    5600 scope.go:122] "RemoveContainer" containerID="f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b"
Nov 03 20:25:41.763458 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:25:41.763421    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b\": container with ID starting with f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b not found: ID does not exist" containerID="f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b"
Nov 03 20:25:41.763527 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.763472    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b"} err="failed to get container status \"f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b\": rpc error: code = NotFound desc = could not find container \"f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b\": container with ID starting with f289c0007b41e55a3317e50dee6983210002d7137591a65a9b3a04625a6ac20b not found: ID does not exist"
Nov 03 20:25:41.790669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.790640    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"]
Nov 03 20:25:41.799328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.799298    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-jbb5g"]
Nov 03 20:25:41.834313 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.834278    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzkmj\" (UniqueName: \"kubernetes.io/projected/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-kube-api-access-gzkmj\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.834313 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.834311    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.834549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:41.834323    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:43.642232 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:43.640658    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" path="/var/lib/kubelet/pods/2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7/volumes"
Nov 03 20:25:44.267122 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.267062    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr" podUID="0deab001-abc3-432e-982f-8b3d0f16008b" containerName="main" containerID="cri-o://571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1" gracePeriod=45
Nov 03 20:25:44.601799 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.601774    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:25:44.678971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.678935    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-home\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-home\") pod \"0deab001-abc3-432e-982f-8b3d0f16008b\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") "
Nov 03 20:25:44.679712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.679047    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0deab001-abc3-432e-982f-8b3d0f16008b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0deab001-abc3-432e-982f-8b3d0f16008b-tls-certs\") pod \"0deab001-abc3-432e-982f-8b3d0f16008b\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") "
Nov 03 20:25:44.679712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.679096    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-model-cache\") pod \"0deab001-abc3-432e-982f-8b3d0f16008b\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") "
Nov 03 20:25:44.679712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.679266    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0deab001-abc3-432e-982f-8b3d0f16008b-kube-api-access-hpbgl\" (UniqueName: \"kubernetes.io/projected/0deab001-abc3-432e-982f-8b3d0f16008b-kube-api-access-hpbgl\") pod \"0deab001-abc3-432e-982f-8b3d0f16008b\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") "
Nov 03 20:25:44.679712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.679288    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-home" pod "0deab001-abc3-432e-982f-8b3d0f16008b" (UID: "0deab001-abc3-432e-982f-8b3d0f16008b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:44.679712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.679316    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-dshm\") pod \"0deab001-abc3-432e-982f-8b3d0f16008b\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") "
Nov 03 20:25:44.679712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.679389    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-tmp-dir\") pod \"0deab001-abc3-432e-982f-8b3d0f16008b\" (UID: \"0deab001-abc3-432e-982f-8b3d0f16008b\") "
Nov 03 20:25:44.680242 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.680202    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:44.681416 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.681386    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-tmp-dir" pod "0deab001-abc3-432e-982f-8b3d0f16008b" (UID: "0deab001-abc3-432e-982f-8b3d0f16008b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:44.681694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.681621    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-model-cache" pod "0deab001-abc3-432e-982f-8b3d0f16008b" (UID: "0deab001-abc3-432e-982f-8b3d0f16008b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:44.682661 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.682638    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-dshm" pod "0deab001-abc3-432e-982f-8b3d0f16008b" (UID: "0deab001-abc3-432e-982f-8b3d0f16008b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:44.684246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.684217    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0deab001-abc3-432e-982f-8b3d0f16008b-kube-api-access-hpbgl" pod "0deab001-abc3-432e-982f-8b3d0f16008b" (UID: "0deab001-abc3-432e-982f-8b3d0f16008b"). InnerVolumeSpecName "kube-api-access-hpbgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:44.684496 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.684471    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0deab001-abc3-432e-982f-8b3d0f16008b-tls-certs" pod "0deab001-abc3-432e-982f-8b3d0f16008b" (UID: "0deab001-abc3-432e-982f-8b3d0f16008b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:44.781561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.781399    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpbgl\" (UniqueName: \"kubernetes.io/projected/0deab001-abc3-432e-982f-8b3d0f16008b-kube-api-access-hpbgl\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:44.781561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.781446    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:44.781561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.781461    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:44.781561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.781483    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0deab001-abc3-432e-982f-8b3d0f16008b-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:44.781561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.781493    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0deab001-abc3-432e-982f-8b3d0f16008b-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:44.789319 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.789287    5600 generic.go:356] "Generic (PLEG): container finished" podID="0deab001-abc3-432e-982f-8b3d0f16008b" containerID="571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1" exitCode=0
Nov 03 20:25:44.789549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.789327    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"
Nov 03 20:25:44.789549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.789360    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr" event={"ID":"0deab001-abc3-432e-982f-8b3d0f16008b","Type":"ContainerDied","Data":"571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1"}
Nov 03 20:25:44.789549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.789405    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr" event={"ID":"0deab001-abc3-432e-982f-8b3d0f16008b","Type":"ContainerDied","Data":"7ee04ac1d20765d3102bb5df3c9779eafac31176038d8888b9376d0f5d1640b8"}
Nov 03 20:25:44.789549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.789426    5600 scope.go:122] "RemoveContainer" containerID="571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1"
Nov 03 20:25:44.804053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.804025    5600 scope.go:122] "RemoveContainer" containerID="571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1"
Nov 03 20:25:44.804719 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:25:44.804693    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1\": container with ID starting with 571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1 not found: ID does not exist" containerID="571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1"
Nov 03 20:25:44.804817 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.804728    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1"} err="failed to get container status \"571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1\": rpc error: code = NotFound desc = could not find container \"571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1\": container with ID starting with 571f8a9805ab5128ab36a40a501115dac9577795df7bed4243200b2e7b608fa1 not found: ID does not exist"
Nov 03 20:25:44.814939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.814913    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"]
Nov 03 20:25:44.818797 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:44.818746    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-ee2323/auth-test-model-ee2323-kserve-86547c9c7c-t5rbr"]
Nov 03 20:25:45.621466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:45.621414    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0deab001-abc3-432e-982f-8b3d0f16008b" path="/var/lib/kubelet/pods/0deab001-abc3-432e-982f-8b3d0f16008b/volumes"
Nov 03 20:25:46.530968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:46.530919    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7" podUID="441b3899-e37a-4c0a-90e4-8cbf1ece2c65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:25:47.024463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:47.018485    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"]
Nov 03 20:25:47.948674 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:47.948626    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z" podUID="06da157a-f7c8-41e6-870a-a630afb5d912" containerName="main" containerID="cri-o://9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e" gracePeriod=45
Nov 03 20:25:48.275201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.275175    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:25:48.399308 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.399275    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06da157a-f7c8-41e6-870a-a630afb5d912-tls-certs\" (UniqueName: \"kubernetes.io/secret/06da157a-f7c8-41e6-870a-a630afb5d912-tls-certs\") pod \"06da157a-f7c8-41e6-870a-a630afb5d912\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") "
Nov 03 20:25:48.399607 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.399583    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06da157a-f7c8-41e6-870a-a630afb5d912-kube-api-access-92d4b\" (UniqueName: \"kubernetes.io/projected/06da157a-f7c8-41e6-870a-a630afb5d912-kube-api-access-92d4b\") pod \"06da157a-f7c8-41e6-870a-a630afb5d912\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") "
Nov 03 20:25:48.399683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.399638    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-dshm\") pod \"06da157a-f7c8-41e6-870a-a630afb5d912\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") "
Nov 03 20:25:48.399790 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.399775    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-tmp-dir\") pod \"06da157a-f7c8-41e6-870a-a630afb5d912\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") "
Nov 03 20:25:48.399853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.399817    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-model-cache\") pod \"06da157a-f7c8-41e6-870a-a630afb5d912\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") "
Nov 03 20:25:48.399915 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.399866    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-home\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-home\") pod \"06da157a-f7c8-41e6-870a-a630afb5d912\" (UID: \"06da157a-f7c8-41e6-870a-a630afb5d912\") "
Nov 03 20:25:48.400122 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.400089    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-tmp-dir" pod "06da157a-f7c8-41e6-870a-a630afb5d912" (UID: "06da157a-f7c8-41e6-870a-a630afb5d912"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:48.400261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.400238    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-model-cache" pod "06da157a-f7c8-41e6-870a-a630afb5d912" (UID: "06da157a-f7c8-41e6-870a-a630afb5d912"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:48.400319 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.400263    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-home" pod "06da157a-f7c8-41e6-870a-a630afb5d912" (UID: "06da157a-f7c8-41e6-870a-a630afb5d912"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:48.400794 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.400766    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:48.400922 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.400797    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:48.400922 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.400812    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:48.402743 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.402704    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06da157a-f7c8-41e6-870a-a630afb5d912-kube-api-access-92d4b" pod "06da157a-f7c8-41e6-870a-a630afb5d912" (UID: "06da157a-f7c8-41e6-870a-a630afb5d912"). InnerVolumeSpecName "kube-api-access-92d4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:48.402876 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.402819    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06da157a-f7c8-41e6-870a-a630afb5d912-tls-certs" pod "06da157a-f7c8-41e6-870a-a630afb5d912" (UID: "06da157a-f7c8-41e6-870a-a630afb5d912"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:48.403659 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.403637    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-dshm" pod "06da157a-f7c8-41e6-870a-a630afb5d912" (UID: "06da157a-f7c8-41e6-870a-a630afb5d912"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:48.502258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.502175    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06da157a-f7c8-41e6-870a-a630afb5d912-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:48.502258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.502208    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92d4b\" (UniqueName: \"kubernetes.io/projected/06da157a-f7c8-41e6-870a-a630afb5d912-kube-api-access-92d4b\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:48.502258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.502221    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06da157a-f7c8-41e6-870a-a630afb5d912-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:48.847413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.847306    5600 generic.go:356] "Generic (PLEG): container finished" podID="06da157a-f7c8-41e6-870a-a630afb5d912" containerID="9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e" exitCode=0
Nov 03 20:25:48.847413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.847363    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z" event={"ID":"06da157a-f7c8-41e6-870a-a630afb5d912","Type":"ContainerDied","Data":"9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e"}
Nov 03 20:25:48.847413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.847374    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"
Nov 03 20:25:48.847413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.847409    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z" event={"ID":"06da157a-f7c8-41e6-870a-a630afb5d912","Type":"ContainerDied","Data":"a396b37ad2ba387a102a2bb6b69ad7829d2e7264968e9d9e0f9b8ed5427694c8"}
Nov 03 20:25:48.847413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.847446    5600 scope.go:122] "RemoveContainer" containerID="9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e"
Nov 03 20:25:48.870367 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.864919    5600 scope.go:122] "RemoveContainer" containerID="9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e"
Nov 03 20:25:48.870367 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:25:48.865256    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e\": container with ID starting with 9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e not found: ID does not exist" containerID="9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e"
Nov 03 20:25:48.870367 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.865284    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e"} err="failed to get container status \"9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e\": rpc error: code = NotFound desc = could not find container \"9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e\": container with ID starting with 9db9bc997ff8f0725004c984e8fb21f2c9a579be6f64c5fd1b09776c3edc3c6e not found: ID does not exist"
Nov 03 20:25:48.892483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.888089    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"]
Nov 03 20:25:48.893771 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:48.893740    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-0ffd89/auth-test-model-0ffd89-kserve-759768c8b5-87q9z"]
Nov 03 20:25:49.630762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:49.630654    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06da157a-f7c8-41e6-870a-a630afb5d912" path="/var/lib/kubelet/pods/06da157a-f7c8-41e6-870a-a630afb5d912/volumes"
Nov 03 20:25:50.873335 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:50.873253    5600 generic.go:356] "Generic (PLEG): container finished" podID="441b3899-e37a-4c0a-90e4-8cbf1ece2c65" containerID="227c7f9079dc2c4381767925d75ea019f246220d1bf7596e03e3b802a2377637" exitCode=0
Nov 03 20:25:50.873758 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:50.873380    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7" event={"ID":"441b3899-e37a-4c0a-90e4-8cbf1ece2c65","Type":"ContainerDied","Data":"227c7f9079dc2c4381767925d75ea019f246220d1bf7596e03e3b802a2377637"}
Nov 03 20:25:50.919897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:50.919866    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:25:51.070562 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.070525    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-kube-api-access-m64dx\" (UniqueName: \"kubernetes.io/projected/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-kube-api-access-m64dx\") pod \"441b3899-e37a-4c0a-90e4-8cbf1ece2c65\" (UID: \"441b3899-e37a-4c0a-90e4-8cbf1ece2c65\") "
Nov 03 20:25:51.070741 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.070600    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-plugins-config-volume\") pod \"441b3899-e37a-4c0a-90e4-8cbf1ece2c65\" (UID: \"441b3899-e37a-4c0a-90e4-8cbf1ece2c65\") "
Nov 03 20:25:51.070976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.070948    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-plugins-config-volume" pod "441b3899-e37a-4c0a-90e4-8cbf1ece2c65" (UID: "441b3899-e37a-4c0a-90e4-8cbf1ece2c65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:51.071192 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.071178    5600 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-plugins-config-volume\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:51.072923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.072901    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-kube-api-access-m64dx" pod "441b3899-e37a-4c0a-90e4-8cbf1ece2c65" (UID: "441b3899-e37a-4c0a-90e4-8cbf1ece2c65"). InnerVolumeSpecName "kube-api-access-m64dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:51.174572 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.172693    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m64dx\" (UniqueName: \"kubernetes.io/projected/441b3899-e37a-4c0a-90e4-8cbf1ece2c65-kube-api-access-m64dx\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:51.754313 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.754262    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k" podUID="d0694eaa-d338-40b8-96e1-0fb4e54be0de" containerName="main" containerID="cri-o://66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1" gracePeriod=45
Nov 03 20:25:51.935871 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.935668    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7" event={"ID":"441b3899-e37a-4c0a-90e4-8cbf1ece2c65","Type":"ContainerDied","Data":"27fe9345bd5212d34cb090dde1ea39e289295fc5eec4ac0a28f71805be7da3a8"}
Nov 03 20:25:51.935871 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.935755    5600 scope.go:122] "RemoveContainer" containerID="227c7f9079dc2c4381767925d75ea019f246220d1bf7596e03e3b802a2377637"
Nov 03 20:25:51.936468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.935962    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"
Nov 03 20:25:51.968091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.968059    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"]
Nov 03 20:25:51.985474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.983111    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-46c34c-7bcc6dd867-86lf7"]
Nov 03 20:25:51.997503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:51.997403    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k" podUID="d0694eaa-d338-40b8-96e1-0fb4e54be0de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.143:8000/ready\": dial tcp 10.134.0.143:8000: connect: connection refused"
Nov 03 20:25:52.019575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.019517    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk" podUID="50c3f058-5ae3-4ad2-be54-af79ec0d4905" containerName="payload-processing" containerID="cri-o://33ec1a50f4d7a98474dec8a91554e4b6bf6d900263ee002189acc5e2357b01ad" gracePeriod=25
Nov 03 20:25:52.183863 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.183837    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.242380    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-dshm\") pod \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") "
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.242547    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0694eaa-d338-40b8-96e1-0fb4e54be0de-kube-api-access-4pkwv\" (UniqueName: \"kubernetes.io/projected/d0694eaa-d338-40b8-96e1-0fb4e54be0de-kube-api-access-4pkwv\") pod \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") "
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.242593    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tls-certs\") pod \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") "
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.242686    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-home\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-home\") pod \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") "
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.242802    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-model-cache\") pod \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") "
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.242920    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tmp-dir\") pod \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\" (UID: \"d0694eaa-d338-40b8-96e1-0fb4e54be0de\") "
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.245518    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-model-cache" pod "d0694eaa-d338-40b8-96e1-0fb4e54be0de" (UID: "d0694eaa-d338-40b8-96e1-0fb4e54be0de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.245601    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-home" pod "d0694eaa-d338-40b8-96e1-0fb4e54be0de" (UID: "d0694eaa-d338-40b8-96e1-0fb4e54be0de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.245834    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tmp-dir" pod "d0694eaa-d338-40b8-96e1-0fb4e54be0de" (UID: "d0694eaa-d338-40b8-96e1-0fb4e54be0de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.246577    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.246600    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:52.247356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.246614    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:52.253921 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.253887    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tls-certs" pod "d0694eaa-d338-40b8-96e1-0fb4e54be0de" (UID: "d0694eaa-d338-40b8-96e1-0fb4e54be0de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:52.261342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.261279    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-dshm" pod "d0694eaa-d338-40b8-96e1-0fb4e54be0de" (UID: "d0694eaa-d338-40b8-96e1-0fb4e54be0de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:52.262490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.262352    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"]
Nov 03 20:25:52.262727 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.262677    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" podUID="19f8d3e0-be03-463c-92a4-27903498fe1e" containerName="istio-proxy" containerID="cri-o://89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d" gracePeriod=30
Nov 03 20:25:52.263626 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.263600    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0694eaa-d338-40b8-96e1-0fb4e54be0de-kube-api-access-4pkwv" pod "d0694eaa-d338-40b8-96e1-0fb4e54be0de" (UID: "d0694eaa-d338-40b8-96e1-0fb4e54be0de"). InnerVolumeSpecName "kube-api-access-4pkwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:52.273668 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.273638    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"]
Nov 03 20:25:52.273906 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.273879    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerName="istio-proxy" containerID="cri-o://eac60e97736482836a2a667c6dae24d1944be3d401d1cef1a6a4c307cd4580c4" gracePeriod=30
Nov 03 20:25:52.281720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.281646    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"]
Nov 03 20:25:52.282002 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.281977    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy" containerID="cri-o://1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48" gracePeriod=30
Nov 03 20:25:52.347862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.347813    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0694eaa-d338-40b8-96e1-0fb4e54be0de-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:52.347862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.347856    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pkwv\" (UniqueName: \"kubernetes.io/projected/d0694eaa-d338-40b8-96e1-0fb4e54be0de-kube-api-access-4pkwv\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:52.348405 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.347873    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0694eaa-d338-40b8-96e1-0fb4e54be0de-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:52.944679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.944647    5600 generic.go:356] "Generic (PLEG): container finished" podID="d0694eaa-d338-40b8-96e1-0fb4e54be0de" containerID="66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1" exitCode=0
Nov 03 20:25:52.945167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.944706    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"
Nov 03 20:25:52.945167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.944737    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k" event={"ID":"d0694eaa-d338-40b8-96e1-0fb4e54be0de","Type":"ContainerDied","Data":"66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1"}
Nov 03 20:25:52.945167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.944910    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k" event={"ID":"d0694eaa-d338-40b8-96e1-0fb4e54be0de","Type":"ContainerDied","Data":"964f648ac1c3823504aa26afd49364ddccd0fbe9ccd53362b65cd883f15434c6"}
Nov 03 20:25:52.945167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.944944    5600 scope.go:122] "RemoveContainer" containerID="66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1"
Nov 03 20:25:52.955788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.955770    5600 scope.go:122] "RemoveContainer" containerID="66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1"
Nov 03 20:25:52.956059 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:25:52.956040    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1\": container with ID starting with 66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1 not found: ID does not exist" containerID="66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1"
Nov 03 20:25:52.956101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.956067    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1"} err="failed to get container status \"66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1\": rpc error: code = NotFound desc = could not find container \"66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1\": container with ID starting with 66324acdce4241c9a09741a2f5160fc358d7755db9e682cf6a59e61159cb69f1 not found: ID does not exist"
Nov 03 20:25:52.969129 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.969100    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"]
Nov 03 20:25:52.975337 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:52.975312    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-cad531/auth-test-model-cad531-kserve-84574d86d5-kcw9k"]
Nov 03 20:25:53.619564 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:53.619524    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441b3899-e37a-4c0a-90e4-8cbf1ece2c65" path="/var/lib/kubelet/pods/441b3899-e37a-4c0a-90e4-8cbf1ece2c65/volumes"
Nov 03 20:25:53.620105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:53.620085    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0694eaa-d338-40b8-96e1-0fb4e54be0de" path="/var/lib/kubelet/pods/d0694eaa-d338-40b8-96e1-0fb4e54be0de/volumes"
Nov 03 20:25:54.165023 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:54.164984    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"]
Nov 03 20:25:54.165552 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:54.165260    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerName="istio-proxy" containerID="cri-o://0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738" gracePeriod=30
Nov 03 20:25:56.062315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.062272    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9" podUID="2f6e46a8-7993-4245-9e36-92669fe70f34" containerName="main" containerID="cri-o://3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338" gracePeriod=45
Nov 03 20:25:56.414833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.414462    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:25:56.424807 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.424772    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk" podUID="50c3f058-5ae3-4ad2-be54-af79ec0d4905" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:25:56.537568 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.537533    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-model-cache\") pod \"2f6e46a8-7993-4245-9e36-92669fe70f34\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") "
Nov 03 20:25:56.538604 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.538579    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6e46a8-7993-4245-9e36-92669fe70f34-kube-api-access-cs8ck\" (UniqueName: \"kubernetes.io/projected/2f6e46a8-7993-4245-9e36-92669fe70f34-kube-api-access-cs8ck\") pod \"2f6e46a8-7993-4245-9e36-92669fe70f34\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") "
Nov 03 20:25:56.578476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.538063    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-model-cache" pod "2f6e46a8-7993-4245-9e36-92669fe70f34" (UID: "2f6e46a8-7993-4245-9e36-92669fe70f34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:56.578476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.575638    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-dshm\") pod \"2f6e46a8-7993-4245-9e36-92669fe70f34\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") "
Nov 03 20:25:56.578476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.575782    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6e46a8-7993-4245-9e36-92669fe70f34-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6e46a8-7993-4245-9e36-92669fe70f34-tls-certs\") pod \"2f6e46a8-7993-4245-9e36-92669fe70f34\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") "
Nov 03 20:25:56.578476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.575846    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-tmp-dir\") pod \"2f6e46a8-7993-4245-9e36-92669fe70f34\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") "
Nov 03 20:25:56.578476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.575896    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-home\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-home\") pod \"2f6e46a8-7993-4245-9e36-92669fe70f34\" (UID: \"2f6e46a8-7993-4245-9e36-92669fe70f34\") "
Nov 03 20:25:56.578476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.576710    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:56.578476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.577047    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-home" pod "2f6e46a8-7993-4245-9e36-92669fe70f34" (UID: "2f6e46a8-7993-4245-9e36-92669fe70f34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:56.600519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.579195    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6e46a8-7993-4245-9e36-92669fe70f34-kube-api-access-cs8ck" pod "2f6e46a8-7993-4245-9e36-92669fe70f34" (UID: "2f6e46a8-7993-4245-9e36-92669fe70f34"). InnerVolumeSpecName "kube-api-access-cs8ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:56.600519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.580482    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-dshm" pod "2f6e46a8-7993-4245-9e36-92669fe70f34" (UID: "2f6e46a8-7993-4245-9e36-92669fe70f34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:56.600519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.580706    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-tmp-dir" pod "2f6e46a8-7993-4245-9e36-92669fe70f34" (UID: "2f6e46a8-7993-4245-9e36-92669fe70f34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:56.609756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.609709    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6e46a8-7993-4245-9e36-92669fe70f34-tls-certs" pod "2f6e46a8-7993-4245-9e36-92669fe70f34" (UID: "2f6e46a8-7993-4245-9e36-92669fe70f34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:56.677680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.677643    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6e46a8-7993-4245-9e36-92669fe70f34-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:56.677680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.677677    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:56.677680 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.677690    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:56.677936 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.677702    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs8ck\" (UniqueName: \"kubernetes.io/projected/2f6e46a8-7993-4245-9e36-92669fe70f34-kube-api-access-cs8ck\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:56.677936 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.677712    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6e46a8-7993-4245-9e36-92669fe70f34-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:56.806627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.806590    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:25:56.807102 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.806842    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" podUID="19f8d3e0-be03-463c-92a4-27903498fe1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:25:56.832223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:56.832188    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"]
Nov 03 20:25:57.031333 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.031177    5600 generic.go:356] "Generic (PLEG): container finished" podID="2f6e46a8-7993-4245-9e36-92669fe70f34" containerID="3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338" exitCode=0
Nov 03 20:25:57.031866 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.031825    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9" event={"ID":"2f6e46a8-7993-4245-9e36-92669fe70f34","Type":"ContainerDied","Data":"3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338"}
Nov 03 20:25:57.032076 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.032060    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9" event={"ID":"2f6e46a8-7993-4245-9e36-92669fe70f34","Type":"ContainerDied","Data":"3dc76d6f02f8ba9af5e9514ebfed9668fe29d7d6e8850cdf427c3ac14d6aeec8"}
Nov 03 20:25:57.032205 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.032190    5600 scope.go:122] "RemoveContainer" containerID="3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338"
Nov 03 20:25:57.032546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.032512    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"
Nov 03 20:25:57.071697 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.071261    5600 scope.go:122] "RemoveContainer" containerID="3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338"
Nov 03 20:25:57.072114 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:25:57.071806    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338\": container with ID starting with 3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338 not found: ID does not exist" containerID="3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338"
Nov 03 20:25:57.072114 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.071844    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338"} err="failed to get container status \"3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338\": rpc error: code = NotFound desc = could not find container \"3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338\": container with ID starting with 3fc13ee1caa80ddf71235dfed376ac2959f48cd7dfe3d85ab65df5c080033338 not found: ID does not exist"
Nov 03 20:25:57.100548 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.100507    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"]
Nov 03 20:25:57.111059 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.111033    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a476f8/auth-test-model-a476f8-kserve-74bf477bb9-vx2w9"]
Nov 03 20:25:57.483087 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.483036    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"]
Nov 03 20:25:57.483837 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.483783    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r" podUID="fcfea225-609d-4768-9ec1-b9dcf38c350f" containerName="maas-api" containerID="cri-o://cc4ce8a7be4f66116cbd94504d86d20757977f21c855e60be61cbc03be8574c3" gracePeriod=30
Nov 03 20:25:57.565539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.556338    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:25:57.565539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.556394    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:25:57.584028 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.583685    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:25:57.584028 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.583742    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:25:57.656499 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.648722    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6e46a8-7993-4245-9e36-92669fe70f34" path="/var/lib/kubelet/pods/2f6e46a8-7993-4245-9e36-92669fe70f34/volumes"
Nov 03 20:25:57.729978 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.729947    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:25:57.876807 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.876687    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-certs" pod "19f8d3e0-be03-463c-92a4-27903498fe1e" (UID: "19f8d3e0-be03-463c-92a4-27903498fe1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.877038 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.877016    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-certs\") pod \"19f8d3e0-be03-463c-92a4-27903498fe1e\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") "
Nov 03 20:25:57.894635 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.894584    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-credential-socket\") pod \"19f8d3e0-be03-463c-92a4-27903498fe1e\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") "
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.943415    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-data\") pod \"19f8d3e0-be03-463c-92a4-27903498fe1e\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") "
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.943527    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-envoy\") pod \"19f8d3e0-be03-463c-92a4-27903498fe1e\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") "
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.943557    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-token\" (UniqueName: \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-token\") pod \"19f8d3e0-be03-463c-92a4-27903498fe1e\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") "
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.943639    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-podinfo\") pod \"19f8d3e0-be03-463c-92a4-27903498fe1e\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") "
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.943828    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-socket\") pod \"19f8d3e0-be03-463c-92a4-27903498fe1e\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") "
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.943861    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-kube-api-access-5w9ln\" (UniqueName: \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-kube-api-access-5w9ln\") pod \"19f8d3e0-be03-463c-92a4-27903498fe1e\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") "
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.943908    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f8d3e0-be03-463c-92a4-27903498fe1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f8d3e0-be03-463c-92a4-27903498fe1e-istiod-ca-cert\") pod \"19f8d3e0-be03-463c-92a4-27903498fe1e\" (UID: \"19f8d3e0-be03-463c-92a4-27903498fe1e\") "
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.944805    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.903225    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-credential-socket" pod "19f8d3e0-be03-463c-92a4-27903498fe1e" (UID: "19f8d3e0-be03-463c-92a4-27903498fe1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.955139    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-socket" pod "19f8d3e0-be03-463c-92a4-27903498fe1e" (UID: "19f8d3e0-be03-463c-92a4-27903498fe1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.957477    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f8d3e0-be03-463c-92a4-27903498fe1e-istiod-ca-cert" pod "19f8d3e0-be03-463c-92a4-27903498fe1e" (UID: "19f8d3e0-be03-463c-92a4-27903498fe1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.985263    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-podinfo" pod "19f8d3e0-be03-463c-92a4-27903498fe1e" (UID: "19f8d3e0-be03-463c-92a4-27903498fe1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.985362    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-envoy" pod "19f8d3e0-be03-463c-92a4-27903498fe1e" (UID: "19f8d3e0-be03-463c-92a4-27903498fe1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.987343    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-kube-api-access-5w9ln" pod "19f8d3e0-be03-463c-92a4-27903498fe1e" (UID: "19f8d3e0-be03-463c-92a4-27903498fe1e"). InnerVolumeSpecName "kube-api-access-5w9ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:57.988300    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-token" pod "19f8d3e0-be03-463c-92a4-27903498fe1e" (UID: "19f8d3e0-be03-463c-92a4-27903498fe1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:58.000579 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.000378    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-data" pod "19f8d3e0-be03-463c-92a4-27903498fe1e" (UID: "19f8d3e0-be03-463c-92a4-27903498fe1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.060901 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.005929    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.063690    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-certs\") pod \"5409036e-08df-4aa4-8245-23575462a0be\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") "
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.063797    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5409036e-08df-4aa4-8245-23575462a0be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5409036e-08df-4aa4-8245-23575462a0be-istiod-ca-cert\") pod \"5409036e-08df-4aa4-8245-23575462a0be\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") "
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.063845    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-istio-token\" (UniqueName: \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-istio-token\") pod \"5409036e-08df-4aa4-8245-23575462a0be\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") "
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.063899    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5409036e-08df-4aa4-8245-23575462a0be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5409036e-08df-4aa4-8245-23575462a0be-istio-podinfo\") pod \"5409036e-08df-4aa4-8245-23575462a0be\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") "
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.063973    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-envoy\") pod \"5409036e-08df-4aa4-8245-23575462a0be\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") "
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.064048    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-credential-socket\") pod \"5409036e-08df-4aa4-8245-23575462a0be\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") "
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.064082    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-socket\") pod \"5409036e-08df-4aa4-8245-23575462a0be\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") "
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.065022    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.065054    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.065093    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.065111    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.065127    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f8d3e0-be03-463c-92a4-27903498fe1e-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.065166    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f8d3e0-be03-463c-92a4-27903498fe1e-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.065185    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w9ln\" (UniqueName: \"kubernetes.io/projected/19f8d3e0-be03-463c-92a4-27903498fe1e-kube-api-access-5w9ln\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.065202    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f8d3e0-be03-463c-92a4-27903498fe1e-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.067597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.065660    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-socket" pod "5409036e-08df-4aa4-8245-23575462a0be" (UID: "5409036e-08df-4aa4-8245-23575462a0be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.084986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.084882    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5409036e-08df-4aa4-8245-23575462a0be-istiod-ca-cert" pod "5409036e-08df-4aa4-8245-23575462a0be" (UID: "5409036e-08df-4aa4-8245-23575462a0be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:58.084986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.084958    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-certs" pod "5409036e-08df-4aa4-8245-23575462a0be" (UID: "5409036e-08df-4aa4-8245-23575462a0be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.086053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.086014    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-credential-socket" pod "5409036e-08df-4aa4-8245-23575462a0be" (UID: "5409036e-08df-4aa4-8245-23575462a0be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.166015 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.165982    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-kube-api-access-rqwwk\" (UniqueName: \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-kube-api-access-rqwwk\") pod \"5409036e-08df-4aa4-8245-23575462a0be\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") "
Nov 03 20:25:58.166780 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.166744    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-data\") pod \"5409036e-08df-4aa4-8245-23575462a0be\" (UID: \"5409036e-08df-4aa4-8245-23575462a0be\") "
Nov 03 20:25:58.169752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.168427    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-data" pod "5409036e-08df-4aa4-8245-23575462a0be" (UID: "5409036e-08df-4aa4-8245-23575462a0be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.169752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.169604    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.169752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.169622    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5409036e-08df-4aa4-8245-23575462a0be-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.169752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.169632    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.169752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.169641    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.170874 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.170771    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-istio-token" pod "5409036e-08df-4aa4-8245-23575462a0be" (UID: "5409036e-08df-4aa4-8245-23575462a0be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:58.176177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.176017    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-envoy" pod "5409036e-08df-4aa4-8245-23575462a0be" (UID: "5409036e-08df-4aa4-8245-23575462a0be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.176476 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.176414    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-kube-api-access-rqwwk" pod "5409036e-08df-4aa4-8245-23575462a0be" (UID: "5409036e-08df-4aa4-8245-23575462a0be"). InnerVolumeSpecName "kube-api-access-rqwwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:58.176923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.176880    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5409036e-08df-4aa4-8245-23575462a0be-istio-podinfo" pod "5409036e-08df-4aa4-8245-23575462a0be" (UID: "5409036e-08df-4aa4-8245-23575462a0be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:58.178511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.177138    5600 generic.go:356] "Generic (PLEG): container finished" podID="19f8d3e0-be03-463c-92a4-27903498fe1e" containerID="89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d" exitCode=0
Nov 03 20:25:58.178511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.177201    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"
Nov 03 20:25:58.178511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.177338    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" event={"ID":"19f8d3e0-be03-463c-92a4-27903498fe1e","Type":"ContainerDied","Data":"89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d"}
Nov 03 20:25:58.178511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.177390    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd" event={"ID":"19f8d3e0-be03-463c-92a4-27903498fe1e","Type":"ContainerDied","Data":"5b5f84b8b096edcc79414d991afabd8c8174732050d27bda5034e783b401b3f2"}
Nov 03 20:25:58.178511 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.177468    5600 scope.go:122] "RemoveContainer" containerID="89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d"
Nov 03 20:25:58.187679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.186855    5600 generic.go:356] "Generic (PLEG): container finished" podID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerID="eac60e97736482836a2a667c6dae24d1944be3d401d1cef1a6a4c307cd4580c4" exitCode=0
Nov 03 20:25:58.187679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.186951    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" event={"ID":"1b05c7a1-d6cf-4963-a997-62c39ca1217b","Type":"ContainerDied","Data":"eac60e97736482836a2a667c6dae24d1944be3d401d1cef1a6a4c307cd4580c4"}
Nov 03 20:25:58.190779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.190058    5600 generic.go:356] "Generic (PLEG): container finished" podID="5409036e-08df-4aa4-8245-23575462a0be" containerID="1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48" exitCode=0
Nov 03 20:25:58.190779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.190242    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" event={"ID":"5409036e-08df-4aa4-8245-23575462a0be","Type":"ContainerDied","Data":"1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48"}
Nov 03 20:25:58.190779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.190287    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk" event={"ID":"5409036e-08df-4aa4-8245-23575462a0be","Type":"ContainerDied","Data":"46ccf309b63d87280ee2ca33eb2bb3e7ef429e588c8d8b235b7cd831ee93968b"}
Nov 03 20:25:58.190779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.190373    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"
Nov 03 20:25:58.197990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.195387    5600 generic.go:356] "Generic (PLEG): container finished" podID="fcfea225-609d-4768-9ec1-b9dcf38c350f" containerID="cc4ce8a7be4f66116cbd94504d86d20757977f21c855e60be61cbc03be8574c3" exitCode=0
Nov 03 20:25:58.197990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.195510    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r" event={"ID":"fcfea225-609d-4768-9ec1-b9dcf38c350f","Type":"ContainerDied","Data":"cc4ce8a7be4f66116cbd94504d86d20757977f21c855e60be61cbc03be8574c3"}
Nov 03 20:25:58.213356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.213329    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:25:58.218773 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.218752    5600 scope.go:122] "RemoveContainer" containerID="89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d"
Nov 03 20:25:58.219294 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:25:58.219270    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d\": container with ID starting with 89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d not found: ID does not exist" containerID="89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d"
Nov 03 20:25:58.219838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.219805    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d"} err="failed to get container status \"89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d\": rpc error: code = NotFound desc = could not find container \"89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d\": container with ID starting with 89b9a0665cf67e1a8a178f7d084058728129bb296a3b8b875bd31896832f813d not found: ID does not exist"
Nov 03 20:25:58.219947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.219937    5600 scope.go:122] "RemoveContainer" containerID="1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48"
Nov 03 20:25:58.230220 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.230156    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"]
Nov 03 20:25:58.236563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.236533    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-hjcpd"]
Nov 03 20:25:58.238923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.238878    5600 scope.go:122] "RemoveContainer" containerID="1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48"
Nov 03 20:25:58.241118 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:25:58.241096    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48\": container with ID starting with 1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48 not found: ID does not exist" containerID="1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48"
Nov 03 20:25:58.241256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.241234    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48"} err="failed to get container status \"1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48\": rpc error: code = NotFound desc = could not find container \"1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48\": container with ID starting with 1ea263e04836f3c977b59453567b512ea549d3c55e4bb92cda38df8683d59d48 not found: ID does not exist"
Nov 03 20:25:58.249509 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.249485    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"]
Nov 03 20:25:58.255364 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.255315    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-752vk"]
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.270275    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-certs\") pod \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") "
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.270366    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-kube-api-access-p2pv8\" (UniqueName: \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-kube-api-access-p2pv8\") pod \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") "
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.270527    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-podinfo\") pod \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") "
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.270574    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-envoy\") pod \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") "
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.270627    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-socket\") pod \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") "
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.270683    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-data\") pod \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") "
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.270726    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istiod-ca-cert\") pod \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") "
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.270787    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-token\" (UniqueName: \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-token\") pod \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") "
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.270909    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-credential-socket\") pod \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\" (UID: \"1b05c7a1-d6cf-4963-a997-62c39ca1217b\") "
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.271568    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.271595    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqwwk\" (UniqueName: \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-kube-api-access-rqwwk\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.271611    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5409036e-08df-4aa4-8245-23575462a0be-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.271625    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5409036e-08df-4aa4-8245-23575462a0be-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.271640    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5409036e-08df-4aa4-8245-23575462a0be-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.272677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.272045    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-credential-socket" pod "1b05c7a1-d6cf-4963-a997-62c39ca1217b" (UID: "1b05c7a1-d6cf-4963-a997-62c39ca1217b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.279376 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.273205    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-certs" pod "1b05c7a1-d6cf-4963-a997-62c39ca1217b" (UID: "1b05c7a1-d6cf-4963-a997-62c39ca1217b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.279376 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.276104    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-kube-api-access-p2pv8" pod "1b05c7a1-d6cf-4963-a997-62c39ca1217b" (UID: "1b05c7a1-d6cf-4963-a997-62c39ca1217b"). InnerVolumeSpecName "kube-api-access-p2pv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:58.280928 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.280896    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-podinfo" pod "1b05c7a1-d6cf-4963-a997-62c39ca1217b" (UID: "1b05c7a1-d6cf-4963-a997-62c39ca1217b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:58.281168 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.281148    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istiod-ca-cert" pod "1b05c7a1-d6cf-4963-a997-62c39ca1217b" (UID: "1b05c7a1-d6cf-4963-a997-62c39ca1217b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:58.282514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.282482    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-data" pod "1b05c7a1-d6cf-4963-a997-62c39ca1217b" (UID: "1b05c7a1-d6cf-4963-a997-62c39ca1217b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.284666 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.284361    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-token" pod "1b05c7a1-d6cf-4963-a997-62c39ca1217b" (UID: "1b05c7a1-d6cf-4963-a997-62c39ca1217b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:58.284666 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.284625    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-socket" pod "1b05c7a1-d6cf-4963-a997-62c39ca1217b" (UID: "1b05c7a1-d6cf-4963-a997-62c39ca1217b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.290724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.290681    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-envoy" pod "1b05c7a1-d6cf-4963-a997-62c39ca1217b" (UID: "1b05c7a1-d6cf-4963-a997-62c39ca1217b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.373556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.373524    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2pv8\" (UniqueName: \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-kube-api-access-p2pv8\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.373745 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.373732    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.373854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.373843    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.373935 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.373924    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.374019 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.374008    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.374105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.374095    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.374195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.374184    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b05c7a1-d6cf-4963-a997-62c39ca1217b-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.374273 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.374263    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.374385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.374372    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b05c7a1-d6cf-4963-a997-62c39ca1217b-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.380315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.380288    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:25:58.476073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.475989    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfea225-609d-4768-9ec1-b9dcf38c350f-kube-api-access-pqrgg\" (UniqueName: \"kubernetes.io/projected/fcfea225-609d-4768-9ec1-b9dcf38c350f-kube-api-access-pqrgg\") pod \"fcfea225-609d-4768-9ec1-b9dcf38c350f\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") "
Nov 03 20:25:58.476073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.476059    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-metrics-certs\") pod \"fcfea225-609d-4768-9ec1-b9dcf38c350f\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") "
Nov 03 20:25:58.476296 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.476100    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcfea225-609d-4768-9ec1-b9dcf38c350f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfea225-609d-4768-9ec1-b9dcf38c350f-openshift-service-ca\") pod \"fcfea225-609d-4768-9ec1-b9dcf38c350f\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") "
Nov 03 20:25:58.476296 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.476121    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-maas-api-tls\") pod \"fcfea225-609d-4768-9ec1-b9dcf38c350f\" (UID: \"fcfea225-609d-4768-9ec1-b9dcf38c350f\") "
Nov 03 20:25:58.476533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.476500    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfea225-609d-4768-9ec1-b9dcf38c350f-openshift-service-ca" pod "fcfea225-609d-4768-9ec1-b9dcf38c350f" (UID: "fcfea225-609d-4768-9ec1-b9dcf38c350f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:58.476685 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.476647    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfea225-609d-4768-9ec1-b9dcf38c350f-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.478485 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.478460    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-metrics-certs" pod "fcfea225-609d-4768-9ec1-b9dcf38c350f" (UID: "fcfea225-609d-4768-9ec1-b9dcf38c350f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:58.478571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.478495    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-maas-api-tls" pod "fcfea225-609d-4768-9ec1-b9dcf38c350f" (UID: "fcfea225-609d-4768-9ec1-b9dcf38c350f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:58.478571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.478527    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfea225-609d-4768-9ec1-b9dcf38c350f-kube-api-access-pqrgg" pod "fcfea225-609d-4768-9ec1-b9dcf38c350f" (UID: "fcfea225-609d-4768-9ec1-b9dcf38c350f"). InnerVolumeSpecName "kube-api-access-pqrgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:58.578046 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.578013    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrgg\" (UniqueName: \"kubernetes.io/projected/fcfea225-609d-4768-9ec1-b9dcf38c350f-kube-api-access-pqrgg\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.578046 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.578045    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.578287 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:58.578057    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfea225-609d-4768-9ec1-b9dcf38c350f-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.799406 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:25:58.799318    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:25:59.205236 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.205207    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"
Nov 03 20:25:59.205792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.205205    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc" event={"ID":"1b05c7a1-d6cf-4963-a997-62c39ca1217b","Type":"ContainerDied","Data":"9a9bbeeefd1062db4dd76e310c1d802af9ed550f1fadc591c191aa957662f501"}
Nov 03 20:25:59.205910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.205839    5600 scope.go:122] "RemoveContainer" containerID="eac60e97736482836a2a667c6dae24d1944be3d401d1cef1a6a4c307cd4580c4"
Nov 03 20:25:59.208867 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.208842    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"
Nov 03 20:25:59.208986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.208864    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r" event={"ID":"fcfea225-609d-4768-9ec1-b9dcf38c350f","Type":"ContainerDied","Data":"d0167fcd36d5b6f53baa91d1ff9c9536dc0bca99d625a77822ee6e18a89ff875"}
Nov 03 20:25:59.257458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.257392    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"]
Nov 03 20:25:59.263745 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.263714    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-tlklc"]
Nov 03 20:25:59.277648 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.277615    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"]
Nov 03 20:25:59.280605 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.280580    5600 scope.go:122] "RemoveContainer" containerID="cc4ce8a7be4f66116cbd94504d86d20757977f21c855e60be61cbc03be8574c3"
Nov 03 20:25:59.281800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.281777    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-cad531-7b78786b67-g244r"]
Nov 03 20:25:59.467651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.467623    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:25:59.512923 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.512752    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-socket\") pod \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") "
Nov 03 20:25:59.513201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.513091    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-socket" pod "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" (UID: "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:59.513319 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.513248    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istiod-ca-cert\") pod \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") "
Nov 03 20:25:59.513695 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.513664    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istiod-ca-cert" pod "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" (UID: "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:59.513798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.513735    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-envoy\") pod \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") "
Nov 03 20:25:59.514222 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.514199    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-credential-socket\") pod \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") "
Nov 03 20:25:59.514328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.514266    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-kube-api-access-zl4vr\" (UniqueName: \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-kube-api-access-zl4vr\") pod \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") "
Nov 03 20:25:59.514397 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.514329    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-podinfo\") pod \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") "
Nov 03 20:25:59.515490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.514419    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-token\" (UniqueName: \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-token\") pod \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") "
Nov 03 20:25:59.515490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.514508    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-data\") pod \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") "
Nov 03 20:25:59.515490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.514605    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-certs\") pod \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\" (UID: \"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c\") "
Nov 03 20:25:59.515490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.515256    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-credential-socket" pod "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" (UID: "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:59.515490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.515287    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-data" pod "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" (UID: "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:59.515490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.515470    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-certs" pod "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" (UID: "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:59.515952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.515390    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:59.515952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.515913    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:59.515952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.515931    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:59.520156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.520125    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-envoy" pod "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" (UID: "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:59.520657 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.520623    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-kube-api-access-zl4vr" pod "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" (UID: "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c"). InnerVolumeSpecName "kube-api-access-zl4vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:59.520866 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.520834    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-podinfo" pod "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" (UID: "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:59.521864 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.521839    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-token" pod "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" (UID: "6ab4207d-a96d-4bc3-bbd6-d97cdac9774c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:59.617847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.617764    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:59.617847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.617803    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:59.617847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.617819    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:59.617847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.617832    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:59.617847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.617846    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4vr\" (UniqueName: \"kubernetes.io/projected/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-kube-api-access-zl4vr\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:59.618124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.617861    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:59.620634 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.620604    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f8d3e0-be03-463c-92a4-27903498fe1e" path="/var/lib/kubelet/pods/19f8d3e0-be03-463c-92a4-27903498fe1e/volumes"
Nov 03 20:25:59.621110 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.621095    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" path="/var/lib/kubelet/pods/1b05c7a1-d6cf-4963-a997-62c39ca1217b/volumes"
Nov 03 20:25:59.621594 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.621580    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5409036e-08df-4aa4-8245-23575462a0be" path="/var/lib/kubelet/pods/5409036e-08df-4aa4-8245-23575462a0be/volumes"
Nov 03 20:25:59.622388 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:25:59.622362    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfea225-609d-4768-9ec1-b9dcf38c350f" path="/var/lib/kubelet/pods/fcfea225-609d-4768-9ec1-b9dcf38c350f/volumes"
Nov 03 20:26:00.221416 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:00.221373    5600 generic.go:356] "Generic (PLEG): container finished" podID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerID="0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738" exitCode=0
Nov 03 20:26:00.221925 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:00.221417    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" event={"ID":"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c","Type":"ContainerDied","Data":"0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738"}
Nov 03 20:26:00.221925 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:00.221483    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"
Nov 03 20:26:00.221925 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:00.221561    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl" event={"ID":"6ab4207d-a96d-4bc3-bbd6-d97cdac9774c","Type":"ContainerDied","Data":"00e213e53304ef11729aaa7394ea40612315c134a4e55fe489ccccb50f0bd8a6"}
Nov 03 20:26:00.221925 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:00.221593    5600 scope.go:122] "RemoveContainer" containerID="0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738"
Nov 03 20:26:00.233540 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:00.233510    5600 scope.go:122] "RemoveContainer" containerID="0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738"
Nov 03 20:26:00.233878 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:00.233857    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738\": container with ID starting with 0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738 not found: ID does not exist" containerID="0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738"
Nov 03 20:26:00.233957 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:00.233885    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738"} err="failed to get container status \"0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738\": rpc error: code = NotFound desc = could not find container \"0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738\": container with ID starting with 0a9866f3863299e18d820f3a75b967e8f26d69842be2e926ebd246aa3114d738 not found: ID does not exist"
Nov 03 20:26:00.250029 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:00.249973    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"]
Nov 03 20:26:00.253006 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:00.252984    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrkkl"]
Nov 03 20:26:01.618551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:01.618519    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" path="/var/lib/kubelet/pods/6ab4207d-a96d-4bc3-bbd6-d97cdac9774c/volumes"
Nov 03 20:26:02.643196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:02.643163    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"]
Nov 03 20:26:02.643770 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:02.643379    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy" containerID="cri-o://04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046" gracePeriod=30
Nov 03 20:26:06.424484 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:06.424399    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk" podUID="50c3f058-5ae3-4ad2-be54-af79ec0d4905" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:26:06.544258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:06.544215    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"]
Nov 03 20:26:06.544800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:06.544751    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerName="istio-proxy" containerID="cri-o://a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346" gracePeriod=30
Nov 03 20:26:06.552084 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:06.552050    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"]
Nov 03 20:26:06.552797 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:06.552560    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" podUID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" containerName="istio-proxy" containerID="cri-o://fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410" gracePeriod=30
Nov 03 20:26:07.948107 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:07.948076    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 20:26:08.081262 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.081220    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-socket\") pod \"9bbda282-0b8b-4133-825d-a962403ea7cd\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") "
Nov 03 20:26:08.081518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.081271    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-podinfo\") pod \"9bbda282-0b8b-4133-825d-a962403ea7cd\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") "
Nov 03 20:26:08.081518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.081304    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-certs\") pod \"9bbda282-0b8b-4133-825d-a962403ea7cd\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") "
Nov 03 20:26:08.081518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.081358    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-envoy\") pod \"9bbda282-0b8b-4133-825d-a962403ea7cd\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") "
Nov 03 20:26:08.081518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.081387    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-token\" (UniqueName: \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-token\") pod \"9bbda282-0b8b-4133-825d-a962403ea7cd\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") "
Nov 03 20:26:08.081518 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.081427    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-credential-socket\") pod \"9bbda282-0b8b-4133-825d-a962403ea7cd\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") "
Nov 03 20:26:08.081819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.081610    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-socket" pod "9bbda282-0b8b-4133-825d-a962403ea7cd" (UID: "9bbda282-0b8b-4133-825d-a962403ea7cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:08.081819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.081633    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bbda282-0b8b-4133-825d-a962403ea7cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bbda282-0b8b-4133-825d-a962403ea7cd-istiod-ca-cert\") pod \"9bbda282-0b8b-4133-825d-a962403ea7cd\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") "
Nov 03 20:26:08.081819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.081668    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-data\") pod \"9bbda282-0b8b-4133-825d-a962403ea7cd\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") "
Nov 03 20:26:08.081819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.081713    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-kube-api-access-x2ccq\" (UniqueName: \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-kube-api-access-x2ccq\") pod \"9bbda282-0b8b-4133-825d-a962403ea7cd\" (UID: \"9bbda282-0b8b-4133-825d-a962403ea7cd\") "
Nov 03 20:26:08.082023 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.081839    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-credential-socket" pod "9bbda282-0b8b-4133-825d-a962403ea7cd" (UID: "9bbda282-0b8b-4133-825d-a962403ea7cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:08.082083 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.082027    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-data" pod "9bbda282-0b8b-4133-825d-a962403ea7cd" (UID: "9bbda282-0b8b-4133-825d-a962403ea7cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:08.082141 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.082113    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbda282-0b8b-4133-825d-a962403ea7cd-istiod-ca-cert" pod "9bbda282-0b8b-4133-825d-a962403ea7cd" (UID: "9bbda282-0b8b-4133-825d-a962403ea7cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:08.082428 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.082309    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-certs" pod "9bbda282-0b8b-4133-825d-a962403ea7cd" (UID: "9bbda282-0b8b-4133-825d-a962403ea7cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:08.082428 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.082361    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.082428 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.082377    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.082428 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.082392    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bbda282-0b8b-4133-825d-a962403ea7cd-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.082428 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.082406    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.087537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.087494    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-token" pod "9bbda282-0b8b-4133-825d-a962403ea7cd" (UID: "9bbda282-0b8b-4133-825d-a962403ea7cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:08.090576 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.090546    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-envoy" pod "9bbda282-0b8b-4133-825d-a962403ea7cd" (UID: "9bbda282-0b8b-4133-825d-a962403ea7cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:08.091201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.091151    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-kube-api-access-x2ccq" pod "9bbda282-0b8b-4133-825d-a962403ea7cd" (UID: "9bbda282-0b8b-4133-825d-a962403ea7cd"). InnerVolumeSpecName "kube-api-access-x2ccq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:08.091201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.091176    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-podinfo" pod "9bbda282-0b8b-4133-825d-a962403ea7cd" (UID: "9bbda282-0b8b-4133-825d-a962403ea7cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:08.184045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.183951    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.184045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.183981    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.184045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.183991    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.184045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.184002    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.184045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.184012    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2ccq\" (UniqueName: \"kubernetes.io/projected/9bbda282-0b8b-4133-825d-a962403ea7cd-kube-api-access-x2ccq\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.312066 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.312029    5600 generic.go:356] "Generic (PLEG): container finished" podID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerID="04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046" exitCode=0
Nov 03 20:26:08.312245 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.312079    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"
Nov 03 20:26:08.312245 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.312117    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" event={"ID":"9bbda282-0b8b-4133-825d-a962403ea7cd","Type":"ContainerDied","Data":"04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046"}
Nov 03 20:26:08.312245 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.312173    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h" event={"ID":"9bbda282-0b8b-4133-825d-a962403ea7cd","Type":"ContainerDied","Data":"4f51e58cecd1de31e829f2ceb38eee02b20bf22c23de4b74e91934721d3e30af"}
Nov 03 20:26:08.312245 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.312204    5600 scope.go:122] "RemoveContainer" containerID="04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046"
Nov 03 20:26:08.325569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.325545    5600 scope.go:122] "RemoveContainer" containerID="04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046"
Nov 03 20:26:08.325901 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:08.325877    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046\": container with ID starting with 04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046 not found: ID does not exist" containerID="04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046"
Nov 03 20:26:08.325996 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.325939    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046"} err="failed to get container status \"04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046\": rpc error: code = NotFound desc = could not find container \"04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046\": container with ID starting with 04171cd8ffec2536f4a2b7c1799457269b202ad4a2dc0b4b8528286cc34de046 not found: ID does not exist"
Nov 03 20:26:08.347424 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.347393    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"]
Nov 03 20:26:08.352756 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:08.352359    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-28w6h"]
Nov 03 20:26:09.618970 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:09.618938    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" path="/var/lib/kubelet/pods/9bbda282-0b8b-4133-825d-a962403ea7cd/volumes"
Nov 03 20:26:09.699815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:09.699781    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"]
Nov 03 20:26:09.700722 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:09.700116    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy" containerID="cri-o://1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb" gracePeriod=30
Nov 03 20:26:10.725592 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:10.725555    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:10.726097 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:10.725646    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:11.863662 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:11.863625    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l" podUID="3c3c6285-c426-4c53-affa-ea450c508260" containerName="main" containerID="cri-o://481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026" gracePeriod=45
Nov 03 20:26:11.939385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:11.939339    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:26:12.030843 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.029972    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:26:12.030843 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.030028    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" podUID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:26:12.054494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.054176    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-envoy\") pod \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") "
Nov 03 20:26:12.054494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.054220    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-kube-api-access-9gq4q\" (UniqueName: \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-kube-api-access-9gq4q\") pod \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") "
Nov 03 20:26:12.054494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.054266    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-data\") pod \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") "
Nov 03 20:26:12.054494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.054297    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-token\" (UniqueName: \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-token\") pod \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") "
Nov 03 20:26:12.054494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.054320    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istiod-ca-cert\") pod \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") "
Nov 03 20:26:12.054494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.054361    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-certs\") pod \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") "
Nov 03 20:26:12.054494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.054414    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-podinfo\") pod \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") "
Nov 03 20:26:12.055088 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.055055    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istiod-ca-cert" pod "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" (UID: "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:12.055183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.055134    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-data" pod "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" (UID: "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.055246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.055221    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-certs" pod "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" (UID: "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.056227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.055410    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-credential-socket\") pod \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") "
Nov 03 20:26:12.056227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.055468    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-socket\") pod \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\" (UID: \"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035\") "
Nov 03 20:26:12.056227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.056192    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-credential-socket" pod "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" (UID: "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.056536 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.056358    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-socket" pod "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" (UID: "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.056779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.056747    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.056779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.056771    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.056918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.056786    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.056918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.056802    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.056918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.056816    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.057786 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.057722    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-podinfo" pod "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" (UID: "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:12.058754 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.058728    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-token" pod "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" (UID: "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.059367 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.059326    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-kube-api-access-9gq4q" pod "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" (UID: "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035"). InnerVolumeSpecName "kube-api-access-9gq4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.059594 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.059572    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-envoy" pod "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" (UID: "25a5589f-5fd8-4f65-88c3-a2ef9dbc2035"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.159101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.159069    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.159101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.159101    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.159378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.159112    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gq4q\" (UniqueName: \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-kube-api-access-9gq4q\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.159378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.159124    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.173824 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.172109    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:26:12.260952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.260912    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-home\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-home\") pod \"3c3c6285-c426-4c53-affa-ea450c508260\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") "
Nov 03 20:26:12.261123 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.261050    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3c6285-c426-4c53-affa-ea450c508260-kube-api-access-sjg77\" (UniqueName: \"kubernetes.io/projected/3c3c6285-c426-4c53-affa-ea450c508260-kube-api-access-sjg77\") pod \"3c3c6285-c426-4c53-affa-ea450c508260\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") "
Nov 03 20:26:12.261123 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.261090    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-dshm\") pod \"3c3c6285-c426-4c53-affa-ea450c508260\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") "
Nov 03 20:26:12.261201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.261122    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-model-cache\") pod \"3c3c6285-c426-4c53-affa-ea450c508260\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") "
Nov 03 20:26:12.262735 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.262681    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-model-cache" pod "3c3c6285-c426-4c53-affa-ea450c508260" (UID: "3c3c6285-c426-4c53-affa-ea450c508260"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.263161 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.263124    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-home" pod "3c3c6285-c426-4c53-affa-ea450c508260" (UID: "3c3c6285-c426-4c53-affa-ea450c508260"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.263777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.263574    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c3c6285-c426-4c53-affa-ea450c508260-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3c6285-c426-4c53-affa-ea450c508260-tls-certs\") pod \"3c3c6285-c426-4c53-affa-ea450c508260\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") "
Nov 03 20:26:12.263777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.263688    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-tmp-dir\") pod \"3c3c6285-c426-4c53-affa-ea450c508260\" (UID: \"3c3c6285-c426-4c53-affa-ea450c508260\") "
Nov 03 20:26:12.264406 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.264187    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-tmp-dir" pod "3c3c6285-c426-4c53-affa-ea450c508260" (UID: "3c3c6285-c426-4c53-affa-ea450c508260"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.264406 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.264344    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.264406 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.264364    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.264406 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.264378    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.265047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.265014    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-dshm" pod "3c3c6285-c426-4c53-affa-ea450c508260" (UID: "3c3c6285-c426-4c53-affa-ea450c508260"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.267556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.267485    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3c6285-c426-4c53-affa-ea450c508260-tls-certs" pod "3c3c6285-c426-4c53-affa-ea450c508260" (UID: "3c3c6285-c426-4c53-affa-ea450c508260"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:12.267641 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.267570    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3c6285-c426-4c53-affa-ea450c508260-kube-api-access-sjg77" pod "3c3c6285-c426-4c53-affa-ea450c508260" (UID: "3c3c6285-c426-4c53-affa-ea450c508260"). InnerVolumeSpecName "kube-api-access-sjg77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.280331 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.280303    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:26:12.356614 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.356574    5600 generic.go:356] "Generic (PLEG): container finished" podID="50c3f058-5ae3-4ad2-be54-af79ec0d4905" containerID="33ec1a50f4d7a98474dec8a91554e4b6bf6d900263ee002189acc5e2357b01ad" exitCode=0
Nov 03 20:26:12.356798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.356655    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk" event={"ID":"50c3f058-5ae3-4ad2-be54-af79ec0d4905","Type":"ContainerDied","Data":"33ec1a50f4d7a98474dec8a91554e4b6bf6d900263ee002189acc5e2357b01ad"}
Nov 03 20:26:12.358592 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.358563    5600 generic.go:356] "Generic (PLEG): container finished" podID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerID="a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346" exitCode=0
Nov 03 20:26:12.358736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.358629    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"
Nov 03 20:26:12.358736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.358712    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" event={"ID":"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035","Type":"ContainerDied","Data":"a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346"}
Nov 03 20:26:12.358860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.358758    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw" event={"ID":"25a5589f-5fd8-4f65-88c3-a2ef9dbc2035","Type":"ContainerDied","Data":"da4b3268092b4acc0664c3a902099a92426cb6397f95d4ce433b35946a004d7b"}
Nov 03 20:26:12.358860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.358790    5600 scope.go:122] "RemoveContainer" containerID="a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346"
Nov 03 20:26:12.360888 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.360867    5600 generic.go:356] "Generic (PLEG): container finished" podID="3c3c6285-c426-4c53-affa-ea450c508260" containerID="481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026" exitCode=0
Nov 03 20:26:12.360992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.360944    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l" event={"ID":"3c3c6285-c426-4c53-affa-ea450c508260","Type":"ContainerDied","Data":"481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026"}
Nov 03 20:26:12.360992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.360972    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l" event={"ID":"3c3c6285-c426-4c53-affa-ea450c508260","Type":"ContainerDied","Data":"b4627b9fe1cbdafdbb41d36a3adc50841d833e1ed7cc3f7dca485fc792fda01d"}
Nov 03 20:26:12.361117 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.361036    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"
Nov 03 20:26:12.366358 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.366331    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjg77\" (UniqueName: \"kubernetes.io/projected/3c3c6285-c426-4c53-affa-ea450c508260-kube-api-access-sjg77\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.366562 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.366364    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3c6285-c426-4c53-affa-ea450c508260-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.366562 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.366380    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3c6285-c426-4c53-affa-ea450c508260-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.373256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.373238    5600 scope.go:122] "RemoveContainer" containerID="a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346"
Nov 03 20:26:12.373539 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:12.373517    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346\": container with ID starting with a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346 not found: ID does not exist" containerID="a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346"
Nov 03 20:26:12.373621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.373557    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346"} err="failed to get container status \"a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346\": rpc error: code = NotFound desc = could not find container \"a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346\": container with ID starting with a0d19e5dcc9a73daa4878a6d200d33e0334a9233aceda3fc3225d52026be2346 not found: ID does not exist"
Nov 03 20:26:12.373621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.373574    5600 scope.go:122] "RemoveContainer" containerID="481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026"
Nov 03 20:26:12.374156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.374123    5600 generic.go:356] "Generic (PLEG): container finished" podID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" containerID="fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410" exitCode=0
Nov 03 20:26:12.374273 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.374256    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" event={"ID":"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338","Type":"ContainerDied","Data":"fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410"}
Nov 03 20:26:12.374323 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.374310    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28" event={"ID":"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338","Type":"ContainerDied","Data":"5d46d029856c37b339eef0ebea09370c4dc357c8e741dd72fdbeddefacc881ca"}
Nov 03 20:26:12.374930 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.374910    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"
Nov 03 20:26:12.379091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.379069    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:26:12.389812 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.389769    5600 scope.go:122] "RemoveContainer" containerID="481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026"
Nov 03 20:26:12.390101 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:12.390066    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026\": container with ID starting with 481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026 not found: ID does not exist" containerID="481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026"
Nov 03 20:26:12.390197 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.390105    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026"} err="failed to get container status \"481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026\": rpc error: code = NotFound desc = could not find container \"481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026\": container with ID starting with 481f072248482da90a461e742e4ef557f3c55bf5fcad88c73821b9f3110f9026 not found: ID does not exist"
Nov 03 20:26:12.390197 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.390127    5600 scope.go:122] "RemoveContainer" containerID="fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410"
Nov 03 20:26:12.399387 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.399169    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"]
Nov 03 20:26:12.405000 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.404979    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-7wfhw"]
Nov 03 20:26:12.417734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.412958    5600 scope.go:122] "RemoveContainer" containerID="fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410"
Nov 03 20:26:12.417734 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:12.416362    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410\": container with ID starting with fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410 not found: ID does not exist" containerID="fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410"
Nov 03 20:26:12.417734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.416398    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410"} err="failed to get container status \"fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410\": rpc error: code = NotFound desc = could not find container \"fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410\": container with ID starting with fa1beeee572c77b4f531fda76385f15154b6592fb4bd23078b1ef3edbbfda410 not found: ID does not exist"
Nov 03 20:26:12.435093 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.434529    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"]
Nov 03 20:26:12.440478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.440426    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-27d3ad/test-model-27d3ad-kserve-596675cdfc-tzf6l"]
Nov 03 20:26:12.467895 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.467862    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-envoy\") pod \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") "
Nov 03 20:26:12.468075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.467924    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-token\" (UniqueName: \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-token\") pod \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") "
Nov 03 20:26:12.468075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.467974    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istiod-ca-cert\") pod \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") "
Nov 03 20:26:12.468075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.468009    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-credential-socket\") pod \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") "
Nov 03 20:26:12.468254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.468151    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-certs\") pod \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") "
Nov 03 20:26:12.468254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.468193    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-socket\") pod \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") "
Nov 03 20:26:12.468370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.468295    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-kube-api-access-mrphg\" (UniqueName: \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-kube-api-access-mrphg\") pod \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") "
Nov 03 20:26:12.468370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.468324    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-credential-socket" pod "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" (UID: "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.468512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.468400    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-certs" pod "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" (UID: "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.468512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.468408    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istiod-ca-cert" pod "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" (UID: "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:12.468512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.468479    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-data\") pod \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") "
Nov 03 20:26:12.468683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.468527    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-podinfo\") pod \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\" (UID: \"cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338\") "
Nov 03 20:26:12.468739 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.468707    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-data" pod "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" (UID: "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.468833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.468807    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-socket" pod "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" (UID: "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.469351 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.469254    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.469351 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.469284    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.469351 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.469299    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.469351 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.469313    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.469351 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.469327    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.470951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.470924    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-token" pod "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" (UID: "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.471075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.470942    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-envoy" pod "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" (UID: "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.471075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.471008    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-kube-api-access-mrphg" pod "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" (UID: "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338"). InnerVolumeSpecName "kube-api-access-mrphg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.471642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.471617    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-podinfo" pod "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" (UID: "cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:12.577158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.575901    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c3f058-5ae3-4ad2-be54-af79ec0d4905-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c3f058-5ae3-4ad2-be54-af79ec0d4905-plugins-config-volume\") pod \"50c3f058-5ae3-4ad2-be54-af79ec0d4905\" (UID: \"50c3f058-5ae3-4ad2-be54-af79ec0d4905\") "
Nov 03 20:26:12.577158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.576014    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c3f058-5ae3-4ad2-be54-af79ec0d4905-kube-api-access-hqjl4\" (UniqueName: \"kubernetes.io/projected/50c3f058-5ae3-4ad2-be54-af79ec0d4905-kube-api-access-hqjl4\") pod \"50c3f058-5ae3-4ad2-be54-af79ec0d4905\" (UID: \"50c3f058-5ae3-4ad2-be54-af79ec0d4905\") "
Nov 03 20:26:12.577158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.576574    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c3f058-5ae3-4ad2-be54-af79ec0d4905-plugins-config-volume" pod "50c3f058-5ae3-4ad2-be54-af79ec0d4905" (UID: "50c3f058-5ae3-4ad2-be54-af79ec0d4905"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:12.577158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.576723    5600 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c3f058-5ae3-4ad2-be54-af79ec0d4905-plugins-config-volume\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.577158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.576744    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.577158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.576758    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.577158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.576771    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrphg\" (UniqueName: \"kubernetes.io/projected/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-kube-api-access-mrphg\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.577158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.576785    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.595572 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.595520    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c3f058-5ae3-4ad2-be54-af79ec0d4905-kube-api-access-hqjl4" pod "50c3f058-5ae3-4ad2-be54-af79ec0d4905" (UID: "50c3f058-5ae3-4ad2-be54-af79ec0d4905"). InnerVolumeSpecName "kube-api-access-hqjl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.678181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.678084    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqjl4\" (UniqueName: \"kubernetes.io/projected/50c3f058-5ae3-4ad2-be54-af79ec0d4905-kube-api-access-hqjl4\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.708625 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.708553    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"]
Nov 03 20:26:12.718465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.717396    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-x2k28"]
Nov 03 20:26:12.730204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.728413    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"]
Nov 03 20:26:12.730204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:12.728771    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7" podUID="87da290c-f102-4519-8977-018368b52685" containerName="maas-api" containerID="cri-o://0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36" gracePeriod=30
Nov 03 20:26:13.205036 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.205003    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:26:13.233669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.233633    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87da290c-f102-4519-8977-018368b52685-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87da290c-f102-4519-8977-018368b52685-openshift-service-ca\") pod \"87da290c-f102-4519-8977-018368b52685\" (UID: \"87da290c-f102-4519-8977-018368b52685\") "
Nov 03 20:26:13.233867 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.233763    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87da290c-f102-4519-8977-018368b52685-kube-api-access-hqqdg\" (UniqueName: \"kubernetes.io/projected/87da290c-f102-4519-8977-018368b52685-kube-api-access-hqqdg\") pod \"87da290c-f102-4519-8977-018368b52685\" (UID: \"87da290c-f102-4519-8977-018368b52685\") "
Nov 03 20:26:13.233867 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.233804    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-maas-api-tls\") pod \"87da290c-f102-4519-8977-018368b52685\" (UID: \"87da290c-f102-4519-8977-018368b52685\") "
Nov 03 20:26:13.233867 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.233841    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-metrics-certs\") pod \"87da290c-f102-4519-8977-018368b52685\" (UID: \"87da290c-f102-4519-8977-018368b52685\") "
Nov 03 20:26:13.234135 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.234070    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87da290c-f102-4519-8977-018368b52685-openshift-service-ca" pod "87da290c-f102-4519-8977-018368b52685" (UID: "87da290c-f102-4519-8977-018368b52685"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:13.234323 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.234290    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87da290c-f102-4519-8977-018368b52685-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:13.236163 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.236138    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87da290c-f102-4519-8977-018368b52685-kube-api-access-hqqdg" pod "87da290c-f102-4519-8977-018368b52685" (UID: "87da290c-f102-4519-8977-018368b52685"). InnerVolumeSpecName "kube-api-access-hqqdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:13.236616 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.236586    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-metrics-certs" pod "87da290c-f102-4519-8977-018368b52685" (UID: "87da290c-f102-4519-8977-018368b52685"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:13.236735 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.236679    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-maas-api-tls" pod "87da290c-f102-4519-8977-018368b52685" (UID: "87da290c-f102-4519-8977-018368b52685"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:13.335404 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.335358    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:13.335404 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.335390    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87da290c-f102-4519-8977-018368b52685-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:13.335404 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.335401    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqqdg\" (UniqueName: \"kubernetes.io/projected/87da290c-f102-4519-8977-018368b52685-kube-api-access-hqqdg\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:13.406719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.404561    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk" event={"ID":"50c3f058-5ae3-4ad2-be54-af79ec0d4905","Type":"ContainerDied","Data":"85f8a0b6b859c92358cc2e54c922f8e683eb381b5739ed67ffe0c7a5edf455c5"}
Nov 03 20:26:13.406719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.404637    5600 scope.go:122] "RemoveContainer" containerID="33ec1a50f4d7a98474dec8a91554e4b6bf6d900263ee002189acc5e2357b01ad"
Nov 03 20:26:13.406719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.404773    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"
Nov 03 20:26:13.421196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.420091    5600 generic.go:356] "Generic (PLEG): container finished" podID="87da290c-f102-4519-8977-018368b52685" containerID="0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36" exitCode=0
Nov 03 20:26:13.421196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.420252    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7" event={"ID":"87da290c-f102-4519-8977-018368b52685","Type":"ContainerDied","Data":"0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36"}
Nov 03 20:26:13.421196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.420302    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7" event={"ID":"87da290c-f102-4519-8977-018368b52685","Type":"ContainerDied","Data":"9c1cd855c53cb52d917f4b687b4c63cef70aa52b54e271c2e34db68ea6689d52"}
Nov 03 20:26:13.421196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.420374    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"
Nov 03 20:26:13.440687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.438845    5600 scope.go:122] "RemoveContainer" containerID="0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36"
Nov 03 20:26:13.462291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.462266    5600 scope.go:122] "RemoveContainer" containerID="0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36"
Nov 03 20:26:13.462628 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:13.462606    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36\": container with ID starting with 0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36 not found: ID does not exist" containerID="0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36"
Nov 03 20:26:13.462719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.462637    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36"} err="failed to get container status \"0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36\": rpc error: code = NotFound desc = could not find container \"0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36\": container with ID starting with 0ead8cdaddce32d3c70c64e7cf359a867c7171661e079a15032278d152bd8e36 not found: ID does not exist"
Nov 03 20:26:13.473336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.473306    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"]
Nov 03 20:26:13.476683 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.476657    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7343-7969d5fd9d-9plqk"]
Nov 03 20:26:13.487705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.487677    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"]
Nov 03 20:26:13.495372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.495344    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d81d18-76d7c87ffc-wv2h7"]
Nov 03 20:26:13.617178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.617140    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" path="/var/lib/kubelet/pods/25a5589f-5fd8-4f65-88c3-a2ef9dbc2035/volumes"
Nov 03 20:26:13.617694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.617675    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3c6285-c426-4c53-affa-ea450c508260" path="/var/lib/kubelet/pods/3c3c6285-c426-4c53-affa-ea450c508260/volumes"
Nov 03 20:26:13.618214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.618183    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c3f058-5ae3-4ad2-be54-af79ec0d4905" path="/var/lib/kubelet/pods/50c3f058-5ae3-4ad2-be54-af79ec0d4905/volumes"
Nov 03 20:26:13.619161 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.619119    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87da290c-f102-4519-8977-018368b52685" path="/var/lib/kubelet/pods/87da290c-f102-4519-8977-018368b52685/volumes"
Nov 03 20:26:13.619757 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:13.619737    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" path="/var/lib/kubelet/pods/cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338/volumes"
Nov 03 20:26:14.534791 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:14.534753    5600 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:14.535193 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:14.534825    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:15.001976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.001942    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:26:15.065235 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065194    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-podinfo\") pod \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") "
Nov 03 20:26:15.065420 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065289    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-socket\") pod \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") "
Nov 03 20:26:15.065420 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065330    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-token\" (UniqueName: \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-token\") pod \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") "
Nov 03 20:26:15.065420 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065411    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-certs\") pod \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") "
Nov 03 20:26:15.065705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065467    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-envoy\") pod \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") "
Nov 03 20:26:15.065705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065569    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-data\") pod \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") "
Nov 03 20:26:15.065705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065604    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-credential-socket\") pod \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") "
Nov 03 20:26:15.065705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065610    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-socket" pod "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" (UID: "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:15.065705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065634    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istiod-ca-cert\") pod \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") "
Nov 03 20:26:15.065976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065735    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-kube-api-access-mbbrv\" (UniqueName: \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-kube-api-access-mbbrv\") pod \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\" (UID: \"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d\") "
Nov 03 20:26:15.065976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065761    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-certs" pod "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" (UID: "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:15.065976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065821    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-data" pod "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" (UID: "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:15.066135 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.065974    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-credential-socket" pod "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" (UID: "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:15.066206 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.066137    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istiod-ca-cert" pod "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" (UID: "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:15.066396 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.066375    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:15.066537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.066402    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:15.066537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.066415    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:15.066537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.066427    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:15.066537 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.066455    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:15.068635 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.068609    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-token" pod "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" (UID: "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:15.068759 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.068742    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-envoy" pod "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" (UID: "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:15.068917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.068899    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-kube-api-access-mbbrv" pod "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" (UID: "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d"). InnerVolumeSpecName "kube-api-access-mbbrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:15.069830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.069809    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-podinfo" pod "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" (UID: "c31b1b2f-fdf9-4bfd-a113-9ceb3503004d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:15.167963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.167926    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:15.167963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.167958    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:15.167963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.167967    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:15.168214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.167976    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbbrv\" (UniqueName: \"kubernetes.io/projected/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d-kube-api-access-mbbrv\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:15.467221 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.467177    5600 generic.go:356] "Generic (PLEG): container finished" podID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerID="1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb" exitCode=0
Nov 03 20:26:15.467221 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.467229    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"
Nov 03 20:26:15.467500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.467248    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" event={"ID":"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d","Type":"ContainerDied","Data":"1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb"}
Nov 03 20:26:15.467500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.467304    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2" event={"ID":"c31b1b2f-fdf9-4bfd-a113-9ceb3503004d","Type":"ContainerDied","Data":"e2ba2a7c1e9bfe266bcc0b0af4174c4d132c36f5b15999fcad1dc61f24442e97"}
Nov 03 20:26:15.467500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.467333    5600 scope.go:122] "RemoveContainer" containerID="1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb"
Nov 03 20:26:15.482076 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.482051    5600 scope.go:122] "RemoveContainer" containerID="1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb"
Nov 03 20:26:15.482579 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:15.482411    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb\": container with ID starting with 1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb not found: ID does not exist" containerID="1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb"
Nov 03 20:26:15.483027 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.482635    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb"} err="failed to get container status \"1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb\": rpc error: code = NotFound desc = could not find container \"1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb\": container with ID starting with 1e0421b18296d10922b65982598e6393e487130da9d3440fe2233c769f1028eb not found: ID does not exist"
Nov 03 20:26:15.497144 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.497095    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"]
Nov 03 20:26:15.500078 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.500052    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-96ms2"]
Nov 03 20:26:15.617136 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:15.617094    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" path="/var/lib/kubelet/pods/c31b1b2f-fdf9-4bfd-a113-9ceb3503004d/volumes"
Nov 03 20:26:16.271618 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:16.271581    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"]
Nov 03 20:26:16.277993 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:16.277962    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"]
Nov 03 20:26:16.291005 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:16.290972    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"]
Nov 03 20:26:20.047172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047134    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"]
Nov 03 20:26:20.047762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047744    5600 cpu_manager.go:414] "containerMap: removing container" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy"
Nov 03 20:26:20.047818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047764    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy"
Nov 03 20:26:20.047818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047777    5600 cpu_manager.go:414] "containerMap: removing container" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy"
Nov 03 20:26:20.047818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047783    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy"
Nov 03 20:26:20.047818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047795    5600 cpu_manager.go:414] "containerMap: removing container" podUID="176d8e7c-0897-4a2a-bb67-cda280625d1e" containerName="maas-api"
Nov 03 20:26:20.047818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047800    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176d8e7c-0897-4a2a-bb67-cda280625d1e" containerName="maas-api"
Nov 03 20:26:20.047818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047808    5600 cpu_manager.go:414] "containerMap: removing container" podUID="a243ff65-42fa-4c4c-97a0-4ed640695dee" containerName="maas-api"
Nov 03 20:26:20.047818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047812    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a243ff65-42fa-4c4c-97a0-4ed640695dee" containerName="maas-api"
Nov 03 20:26:20.047818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047821    5600 cpu_manager.go:414] "containerMap: removing container" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047825    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047833    5600 cpu_manager.go:414] "containerMap: removing container" podUID="87da290c-f102-4519-8977-018368b52685" containerName="maas-api"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047837    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87da290c-f102-4519-8977-018368b52685" containerName="maas-api"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047847    5600 cpu_manager.go:414] "containerMap: removing container" podUID="0deab001-abc3-432e-982f-8b3d0f16008b" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047851    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0deab001-abc3-432e-982f-8b3d0f16008b" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047862    5600 cpu_manager.go:414] "containerMap: removing container" podUID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047866    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047873    5600 cpu_manager.go:414] "containerMap: removing container" podUID="441b3899-e37a-4c0a-90e4-8cbf1ece2c65" containerName="payload-pre-processing"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047878    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441b3899-e37a-4c0a-90e4-8cbf1ece2c65" containerName="payload-pre-processing"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047885    5600 cpu_manager.go:414] "containerMap: removing container" podUID="3c3c6285-c426-4c53-affa-ea450c508260" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047889    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3c6285-c426-4c53-affa-ea450c508260" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047897    5600 cpu_manager.go:414] "containerMap: removing container" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047901    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047910    5600 cpu_manager.go:414] "containerMap: removing container" podUID="50c3f058-5ae3-4ad2-be54-af79ec0d4905" containerName="payload-processing"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047915    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c3f058-5ae3-4ad2-be54-af79ec0d4905" containerName="payload-processing"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047924    5600 cpu_manager.go:414] "containerMap: removing container" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047929    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047935    5600 cpu_manager.go:414] "containerMap: removing container" podUID="06da157a-f7c8-41e6-870a-a630afb5d912" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047939    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06da157a-f7c8-41e6-870a-a630afb5d912" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047948    5600 cpu_manager.go:414] "containerMap: removing container" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047951    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047957    5600 cpu_manager.go:414] "containerMap: removing container" podUID="fcfea225-609d-4768-9ec1-b9dcf38c350f" containerName="maas-api"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047961    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfea225-609d-4768-9ec1-b9dcf38c350f" containerName="maas-api"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047969    5600 cpu_manager.go:414] "containerMap: removing container" podUID="2f6e46a8-7993-4245-9e36-92669fe70f34" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047974    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6e46a8-7993-4245-9e36-92669fe70f34" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047985    5600 cpu_manager.go:414] "containerMap: removing container" podUID="19f8d3e0-be03-463c-92a4-27903498fe1e" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047990    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f8d3e0-be03-463c-92a4-27903498fe1e" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.047997    5600 cpu_manager.go:414] "containerMap: removing container" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048002    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048015    5600 cpu_manager.go:414] "containerMap: removing container" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048020    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048027    5600 cpu_manager.go:414] "containerMap: removing container" podUID="d0694eaa-d338-40b8-96e1-0fb4e54be0de" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048031    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0694eaa-d338-40b8-96e1-0fb4e54be0de" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048037    5600 cpu_manager.go:414] "containerMap: removing container" podUID="2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048042    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048114    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0b36e6-38af-4c60-a4a1-3f3c0ee8b338" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048121    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="441b3899-e37a-4c0a-90e4-8cbf1ece2c65" containerName="payload-pre-processing"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048128    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3c6285-c426-4c53-affa-ea450c508260" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048133    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31b1b2f-fdf9-4bfd-a113-9ceb3503004d" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048139    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c3f058-5ae3-4ad2-be54-af79ec0d4905" containerName="payload-processing"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048144    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab4207d-a96d-4bc3-bbd6-d97cdac9774c" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048150    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="06da157a-f7c8-41e6-870a-a630afb5d912" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048157    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="c544766f-9820-441d-9564-647d963fa1ea" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048162    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfea225-609d-4768-9ec1-b9dcf38c350f" containerName="maas-api"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048167    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6e46a8-7993-4245-9e36-92669fe70f34" containerName="main"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048172    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f8d3e0-be03-463c-92a4-27903498fe1e" containerName="istio-proxy"
Nov 03 20:26:20.048150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048178    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a5589f-5fd8-4f65-88c3-a2ef9dbc2035" containerName="istio-proxy"
Nov 03 20:26:20.049910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048184    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b05c7a1-d6cf-4963-a997-62c39ca1217b" containerName="istio-proxy"
Nov 03 20:26:20.049910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048189    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0694eaa-d338-40b8-96e1-0fb4e54be0de" containerName="main"
Nov 03 20:26:20.049910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048195    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a71e36d-e50f-4c6e-ba07-fa2fb1cc2ee7" containerName="istio-proxy"
Nov 03 20:26:20.049910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048200    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bbda282-0b8b-4133-825d-a962403ea7cd" containerName="istio-proxy"
Nov 03 20:26:20.049910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048204    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="5409036e-08df-4aa4-8245-23575462a0be" containerName="istio-proxy"
Nov 03 20:26:20.049910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048210    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="176d8e7c-0897-4a2a-bb67-cda280625d1e" containerName="maas-api"
Nov 03 20:26:20.049910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048215    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="a243ff65-42fa-4c4c-97a0-4ed640695dee" containerName="maas-api"
Nov 03 20:26:20.049910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048219    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d7ec61-4c37-4cfc-80fe-102b094dfe0c" containerName="istio-proxy"
Nov 03 20:26:20.049910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048223    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="87da290c-f102-4519-8977-018368b52685" containerName="maas-api"
Nov 03 20:26:20.049910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.048230    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="0deab001-abc3-432e-982f-8b3d0f16008b" containerName="main"
Nov 03 20:26:20.053193 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.053175    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.062618 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.061502    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b55105-openshift-default-dockercfg-b42xt\""
Nov 03 20:26:20.063662 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.063630    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"]
Nov 03 20:26:20.143363 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.143317    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-credential-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.143608 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.143415    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da904e7b-25ba-4397-9841-61cccacfc4ae-istiod-ca-cert\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.143608 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.143522    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.143608 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.143576    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-data\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.143803 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.143654    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-token\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.143803 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.143703    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-podinfo\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.143803 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.143768    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-certs\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.143964 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.143808    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-envoy\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.143964 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.143882    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhv8\" (UniqueName: \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-kube-api-access-wzhv8\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.263213    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-credential-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.263300    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da904e7b-25ba-4397-9841-61cccacfc4ae-istiod-ca-cert\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.263364    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.263411    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-data\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.263463    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-token\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.263497    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-podinfo\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.263554    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-certs\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.263587    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-envoy\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.263652    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhv8\" (UniqueName: \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-kube-api-access-wzhv8\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.264382    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-credential-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.265022    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da904e7b-25ba-4397-9841-61cccacfc4ae-istiod-ca-cert\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.265334    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.269503 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.265675    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-data\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.286078 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.286044    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-certs\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.317516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.314116    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-podinfo\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.330519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.318527    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhv8\" (UniqueName: \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-kube-api-access-wzhv8\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.330519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.318981    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-token\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.330519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.319364    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-envoy\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.371815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.371776    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:20.675372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.675318    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"]
Nov 03 20:26:20.677428 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:26:20.677383    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda904e7b_25ba_4397_9841_61cccacfc4ae.slice/crio-4c04493906cb2d31529eeacd8856f7b6f50bbb65ab0e3d25c87dd8e131d5a3d5 WatchSource:0}: Error finding container 4c04493906cb2d31529eeacd8856f7b6f50bbb65ab0e3d25c87dd8e131d5a3d5: Status 404 returned error can't find the container with id 4c04493906cb2d31529eeacd8856f7b6f50bbb65ab0e3d25c87dd8e131d5a3d5
Nov 03 20:26:20.680267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.680231    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:26:20.680390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.680333    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:26:20.680390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.680381    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:26:20.947515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.946572    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"]
Nov 03 20:26:20.947515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:20.947151    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerName="istio-proxy" containerID="cri-o://5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca" gracePeriod=30
Nov 03 20:26:21.530613 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:21.530571    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h" event={"ID":"da904e7b-25ba-4397-9841-61cccacfc4ae","Type":"ContainerStarted","Data":"715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972"}
Nov 03 20:26:21.531139 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:21.530634    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h" event={"ID":"da904e7b-25ba-4397-9841-61cccacfc4ae","Type":"ContainerStarted","Data":"4c04493906cb2d31529eeacd8856f7b6f50bbb65ab0e3d25c87dd8e131d5a3d5"}
Nov 03 20:26:21.612941 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:21.612837    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h" podStartSLOduration=1.612816752 podStartE2EDuration="1.612816752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:21.611834322 +0000 UTC m=+2764.124656104" watchObservedRunningTime="2026-11-03 20:26:21.612816752 +0000 UTC m=+2764.125638535"
Nov 03 20:26:22.372472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:22.372419    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:22.374231 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:22.374205    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 20:26:22.374388 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:22.374257    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h" podUID="da904e7b-25ba-4397-9841-61cccacfc4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 20:26:23.143500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.138761    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"]
Nov 03 20:26:23.143500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.139038    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2" podUID="e0121c77-f76c-4041-8dc3-707f115f5bcb" containerName="maas-api" containerID="cri-o://bddf5e8552bcc3c12b0a21af5a97ee23d9954aa1e2ed74467147e6714cdcf653" gracePeriod=30
Nov 03 20:26:23.398920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.372481    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 20:26:23.398920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.372545    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h" podUID="da904e7b-25ba-4397-9841-61cccacfc4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 20:26:23.718453 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.716702    5600 generic.go:356] "Generic (PLEG): container finished" podID="e0121c77-f76c-4041-8dc3-707f115f5bcb" containerID="bddf5e8552bcc3c12b0a21af5a97ee23d9954aa1e2ed74467147e6714cdcf653" exitCode=0
Nov 03 20:26:23.718453 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.716850    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2" event={"ID":"e0121c77-f76c-4041-8dc3-707f115f5bcb","Type":"ContainerDied","Data":"bddf5e8552bcc3c12b0a21af5a97ee23d9954aa1e2ed74467147e6714cdcf653"}
Nov 03 20:26:23.734413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.734385    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:26:23.809849 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.809807    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0121c77-f76c-4041-8dc3-707f115f5bcb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0121c77-f76c-4041-8dc3-707f115f5bcb-openshift-service-ca\") pod \"e0121c77-f76c-4041-8dc3-707f115f5bcb\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") "
Nov 03 20:26:23.813678 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.813648    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-maas-api-tls\") pod \"e0121c77-f76c-4041-8dc3-707f115f5bcb\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") "
Nov 03 20:26:23.813982 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.813645    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0121c77-f76c-4041-8dc3-707f115f5bcb-openshift-service-ca" pod "e0121c77-f76c-4041-8dc3-707f115f5bcb" (UID: "e0121c77-f76c-4041-8dc3-707f115f5bcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:23.813982 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.813968    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0121c77-f76c-4041-8dc3-707f115f5bcb-kube-api-access-rbj8k\" (UniqueName: \"kubernetes.io/projected/e0121c77-f76c-4041-8dc3-707f115f5bcb-kube-api-access-rbj8k\") pod \"e0121c77-f76c-4041-8dc3-707f115f5bcb\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") "
Nov 03 20:26:23.814569 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.814199    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-metrics-certs\") pod \"e0121c77-f76c-4041-8dc3-707f115f5bcb\" (UID: \"e0121c77-f76c-4041-8dc3-707f115f5bcb\") "
Nov 03 20:26:23.815538 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.815274    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0121c77-f76c-4041-8dc3-707f115f5bcb-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:23.818162 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.818127    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-maas-api-tls" pod "e0121c77-f76c-4041-8dc3-707f115f5bcb" (UID: "e0121c77-f76c-4041-8dc3-707f115f5bcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:23.818308 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.818287    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-metrics-certs" pod "e0121c77-f76c-4041-8dc3-707f115f5bcb" (UID: "e0121c77-f76c-4041-8dc3-707f115f5bcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:23.818308 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.818287    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0121c77-f76c-4041-8dc3-707f115f5bcb-kube-api-access-rbj8k" pod "e0121c77-f76c-4041-8dc3-707f115f5bcb" (UID: "e0121c77-f76c-4041-8dc3-707f115f5bcb"). InnerVolumeSpecName "kube-api-access-rbj8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:23.916288 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.916253    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:23.916288 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.916290    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbj8k\" (UniqueName: \"kubernetes.io/projected/e0121c77-f76c-4041-8dc3-707f115f5bcb-kube-api-access-rbj8k\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:23.916556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:23.916304    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0121c77-f76c-4041-8dc3-707f115f5bcb-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:24.372294 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:24.372263    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 20:26:24.372706 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:24.372315    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h" podUID="da904e7b-25ba-4397-9841-61cccacfc4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 20:26:24.748290 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:24.748120    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2" event={"ID":"e0121c77-f76c-4041-8dc3-707f115f5bcb","Type":"ContainerDied","Data":"6d31c60046043b2237269f4452fdcb5a4417fd2b143d5cd60db91004c3f6a17b"}
Nov 03 20:26:24.748290 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:24.748203    5600 scope.go:122] "RemoveContainer" containerID="bddf5e8552bcc3c12b0a21af5a97ee23d9954aa1e2ed74467147e6714cdcf653"
Nov 03 20:26:24.748661 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:24.748361    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"
Nov 03 20:26:24.748734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:24.748692    5600 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:24.748798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:24.748742    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:24.773863 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:24.773830    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"]
Nov 03 20:26:24.778883 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:24.778854    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a476f8-86b8c9bfff-sjjk2"]
Nov 03 20:26:25.373330 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:25.373285    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 20:26:25.373866 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:25.373361    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h" podUID="da904e7b-25ba-4397-9841-61cccacfc4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 20:26:25.618381 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:25.618347    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0121c77-f76c-4041-8dc3-707f115f5bcb" path="/var/lib/kubelet/pods/e0121c77-f76c-4041-8dc3-707f115f5bcb/volumes"
Nov 03 20:26:26.372635 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.372543    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 20:26:26.372635 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.372619    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h" podUID="da904e7b-25ba-4397-9841-61cccacfc4ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 20:26:26.746243 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.746216    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:26:26.763219 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.763180    5600 generic.go:356] "Generic (PLEG): container finished" podID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerID="5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca" exitCode=0
Nov 03 20:26:26.763342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.763227    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" event={"ID":"5789c64e-f731-4af7-96c0-f044fd64aaf6","Type":"ContainerDied","Data":"5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca"}
Nov 03 20:26:26.763342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.763228    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"
Nov 03 20:26:26.763342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.763284    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp" event={"ID":"5789c64e-f731-4af7-96c0-f044fd64aaf6","Type":"ContainerDied","Data":"8ae1e077da559a0ce02c919941f9d9c5ecd9cf8ba7c87ea39ca6f30f2ccf100d"}
Nov 03 20:26:26.763342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.763315    5600 scope.go:122] "RemoveContainer" containerID="5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca"
Nov 03 20:26:26.773422 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.773400    5600 scope.go:122] "RemoveContainer" containerID="5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca"
Nov 03 20:26:26.773706 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:26.773689    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca\": container with ID starting with 5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca not found: ID does not exist" containerID="5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca"
Nov 03 20:26:26.773775 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.773716    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca"} err="failed to get container status \"5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca\": rpc error: code = NotFound desc = could not find container \"5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca\": container with ID starting with 5b181fd2a13e0e6452e7801cec24320919324d738a72daabc836ff45c6dcf0ca not found: ID does not exist"
Nov 03 20:26:26.865703 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.865667    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-podinfo\") pod \"5789c64e-f731-4af7-96c0-f044fd64aaf6\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") "
Nov 03 20:26:26.865905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.865716    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-data\") pod \"5789c64e-f731-4af7-96c0-f044fd64aaf6\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") "
Nov 03 20:26:26.865905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.865741    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5789c64e-f731-4af7-96c0-f044fd64aaf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5789c64e-f731-4af7-96c0-f044fd64aaf6-istiod-ca-cert\") pod \"5789c64e-f731-4af7-96c0-f044fd64aaf6\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") "
Nov 03 20:26:26.865905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.865786    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-certs\") pod \"5789c64e-f731-4af7-96c0-f044fd64aaf6\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") "
Nov 03 20:26:26.865905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.865814    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-socket\") pod \"5789c64e-f731-4af7-96c0-f044fd64aaf6\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") "
Nov 03 20:26:26.865905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.865888    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-credential-socket\") pod \"5789c64e-f731-4af7-96c0-f044fd64aaf6\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") "
Nov 03 20:26:26.866196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866007    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-data" pod "5789c64e-f731-4af7-96c0-f044fd64aaf6" (UID: "5789c64e-f731-4af7-96c0-f044fd64aaf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.866196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866061    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-token\" (UniqueName: \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-token\") pod \"5789c64e-f731-4af7-96c0-f044fd64aaf6\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") "
Nov 03 20:26:26.866196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866104    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-kube-api-access-f7zg6\" (UniqueName: \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-kube-api-access-f7zg6\") pod \"5789c64e-f731-4af7-96c0-f044fd64aaf6\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") "
Nov 03 20:26:26.866196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866172    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-envoy\") pod \"5789c64e-f731-4af7-96c0-f044fd64aaf6\" (UID: \"5789c64e-f731-4af7-96c0-f044fd64aaf6\") "
Nov 03 20:26:26.866407 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866217    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-certs" pod "5789c64e-f731-4af7-96c0-f044fd64aaf6" (UID: "5789c64e-f731-4af7-96c0-f044fd64aaf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.866407 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866317    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-credential-socket" pod "5789c64e-f731-4af7-96c0-f044fd64aaf6" (UID: "5789c64e-f731-4af7-96c0-f044fd64aaf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.866559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866491    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5789c64e-f731-4af7-96c0-f044fd64aaf6-istiod-ca-cert" pod "5789c64e-f731-4af7-96c0-f044fd64aaf6" (UID: "5789c64e-f731-4af7-96c0-f044fd64aaf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:26.866559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866510    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-socket" pod "5789c64e-f731-4af7-96c0-f044fd64aaf6" (UID: "5789c64e-f731-4af7-96c0-f044fd64aaf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.866945 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866711    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.866945 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866726    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.866945 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866735    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5789c64e-f731-4af7-96c0-f044fd64aaf6-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.866945 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866744    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.866945 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.866753    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.868404 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.868379    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-envoy" pod "5789c64e-f731-4af7-96c0-f044fd64aaf6" (UID: "5789c64e-f731-4af7-96c0-f044fd64aaf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.868604 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.868425    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-podinfo" pod "5789c64e-f731-4af7-96c0-f044fd64aaf6" (UID: "5789c64e-f731-4af7-96c0-f044fd64aaf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:26.868857 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.868840    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-kube-api-access-f7zg6" pod "5789c64e-f731-4af7-96c0-f044fd64aaf6" (UID: "5789c64e-f731-4af7-96c0-f044fd64aaf6"). InnerVolumeSpecName "kube-api-access-f7zg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:26.868917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.868852    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-token" pod "5789c64e-f731-4af7-96c0-f044fd64aaf6" (UID: "5789c64e-f731-4af7-96c0-f044fd64aaf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:26.968415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.968328    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.968415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.968366    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.968415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.968382    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7zg6\" (UniqueName: \"kubernetes.io/projected/5789c64e-f731-4af7-96c0-f044fd64aaf6-kube-api-access-f7zg6\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.968415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:26.968396    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5789c64e-f731-4af7-96c0-f044fd64aaf6-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:27.096175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:27.096135    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"]
Nov 03 20:26:27.103922 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:27.103888    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-qxntp"]
Nov 03 20:26:27.459809 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:27.459760    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:27.460251 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:27.460222    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:27.461234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:27.461213    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:26:27.621466 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:27.621414    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" path="/var/lib/kubelet/pods/5789c64e-f731-4af7-96c0-f044fd64aaf6/volumes"
Nov 03 20:26:28.795976 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:28.795943    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:26:30.374193 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:30.374152    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"]
Nov 03 20:26:30.374768 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:30.374537    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerName="istio-proxy" containerID="cri-o://97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4" gracePeriod=30
Nov 03 20:26:30.413546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:30.413517    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"]
Nov 03 20:26:30.414346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:30.413878    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" podUID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" containerName="istio-proxy" containerID="cri-o://c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02" gracePeriod=30
Nov 03 20:26:31.301099 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.301057    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main" containerID="cri-o://0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef" gracePeriod=45
Nov 03 20:26:31.306950 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.306544    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main" containerID="cri-o://6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462" gracePeriod=45
Nov 03 20:26:31.332507 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.332469    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main" containerID="cri-o://0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b" gracePeriod=45
Nov 03 20:26:31.667271 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.667241    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:26:31.672318 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.671677    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:26:31.701470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.700624    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d1e6564-0a1c-40d3-b800-665251906b46-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1e6564-0a1c-40d3-b800-665251906b46-tls-certs\") pod \"5d1e6564-0a1c-40d3-b800-665251906b46\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") "
Nov 03 20:26:31.701470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.700715    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-home\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-home\") pod \"424be2c2-94d5-42d9-b434-d2bdaa809e85\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") "
Nov 03 20:26:31.701470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.700757    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424be2c2-94d5-42d9-b434-d2bdaa809e85-tls-certs\" (UniqueName: \"kubernetes.io/secret/424be2c2-94d5-42d9-b434-d2bdaa809e85-tls-certs\") pod \"424be2c2-94d5-42d9-b434-d2bdaa809e85\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") "
Nov 03 20:26:31.701470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.700803    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424be2c2-94d5-42d9-b434-d2bdaa809e85-kube-api-access-4x6r6\" (UniqueName: \"kubernetes.io/projected/424be2c2-94d5-42d9-b434-d2bdaa809e85-kube-api-access-4x6r6\") pod \"424be2c2-94d5-42d9-b434-d2bdaa809e85\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") "
Nov 03 20:26:31.701470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.700853    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-dshm\") pod \"424be2c2-94d5-42d9-b434-d2bdaa809e85\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") "
Nov 03 20:26:31.701470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.700910    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-model-cache\") pod \"424be2c2-94d5-42d9-b434-d2bdaa809e85\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") "
Nov 03 20:26:31.701991 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.701942    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-home" pod "424be2c2-94d5-42d9-b434-d2bdaa809e85" (UID: "424be2c2-94d5-42d9-b434-d2bdaa809e85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.702226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.702205    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1e6564-0a1c-40d3-b800-665251906b46-kube-api-access-cdr76\" (UniqueName: \"kubernetes.io/projected/5d1e6564-0a1c-40d3-b800-665251906b46-kube-api-access-cdr76\") pod \"5d1e6564-0a1c-40d3-b800-665251906b46\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") "
Nov 03 20:26:31.702602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.702585    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-tmp-dir\") pod \"424be2c2-94d5-42d9-b434-d2bdaa809e85\" (UID: \"424be2c2-94d5-42d9-b434-d2bdaa809e85\") "
Nov 03 20:26:31.702702 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.702327    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-model-cache" pod "424be2c2-94d5-42d9-b434-d2bdaa809e85" (UID: "424be2c2-94d5-42d9-b434-d2bdaa809e85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.702702 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.702679    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-model-cache\") pod \"5d1e6564-0a1c-40d3-b800-665251906b46\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") "
Nov 03 20:26:31.702829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.702718    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-tmp-dir\") pod \"5d1e6564-0a1c-40d3-b800-665251906b46\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") "
Nov 03 20:26:31.702829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.702753    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-home\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-home\") pod \"5d1e6564-0a1c-40d3-b800-665251906b46\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") "
Nov 03 20:26:31.702829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.702787    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-dshm\") pod \"5d1e6564-0a1c-40d3-b800-665251906b46\" (UID: \"5d1e6564-0a1c-40d3-b800-665251906b46\") "
Nov 03 20:26:31.702993 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.702896    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-tmp-dir" pod "424be2c2-94d5-42d9-b434-d2bdaa809e85" (UID: "424be2c2-94d5-42d9-b434-d2bdaa809e85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.702993 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.702927    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-model-cache" pod "5d1e6564-0a1c-40d3-b800-665251906b46" (UID: "5d1e6564-0a1c-40d3-b800-665251906b46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.703163 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.703085    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-tmp-dir" pod "5d1e6564-0a1c-40d3-b800-665251906b46" (UID: "5d1e6564-0a1c-40d3-b800-665251906b46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.703163 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.703098    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-home" pod "5d1e6564-0a1c-40d3-b800-665251906b46" (UID: "5d1e6564-0a1c-40d3-b800-665251906b46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.703525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.703505    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.703617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.703527    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.703617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.703539    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.703617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.703550    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.703617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.703564    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.703617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.703576    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.703800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.703708    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:26:31.705810 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.705770    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1e6564-0a1c-40d3-b800-665251906b46-tls-certs" pod "5d1e6564-0a1c-40d3-b800-665251906b46" (UID: "5d1e6564-0a1c-40d3-b800-665251906b46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:31.706965 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.706931    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-dshm" pod "424be2c2-94d5-42d9-b434-d2bdaa809e85" (UID: "424be2c2-94d5-42d9-b434-d2bdaa809e85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.707053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.707017    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-dshm" pod "5d1e6564-0a1c-40d3-b800-665251906b46" (UID: "5d1e6564-0a1c-40d3-b800-665251906b46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.707192 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.707161    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424be2c2-94d5-42d9-b434-d2bdaa809e85-tls-certs" pod "424be2c2-94d5-42d9-b434-d2bdaa809e85" (UID: "424be2c2-94d5-42d9-b434-d2bdaa809e85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:31.711234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.711196    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1e6564-0a1c-40d3-b800-665251906b46-kube-api-access-cdr76" pod "5d1e6564-0a1c-40d3-b800-665251906b46" (UID: "5d1e6564-0a1c-40d3-b800-665251906b46"). InnerVolumeSpecName "kube-api-access-cdr76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:31.721932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.721885    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424be2c2-94d5-42d9-b434-d2bdaa809e85-kube-api-access-4x6r6" pod "424be2c2-94d5-42d9-b434-d2bdaa809e85" (UID: "424be2c2-94d5-42d9-b434-d2bdaa809e85"). InnerVolumeSpecName "kube-api-access-4x6r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:31.803498 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.803427    5600 generic.go:356] "Generic (PLEG): container finished" podID="427c352d-209c-43fd-9afd-0a0209d94893" containerID="0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b" exitCode=0
Nov 03 20:26:31.803647 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.803568    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" event={"ID":"427c352d-209c-43fd-9afd-0a0209d94893","Type":"ContainerDied","Data":"0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b"}
Nov 03 20:26:31.803647 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.803599    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql" event={"ID":"427c352d-209c-43fd-9afd-0a0209d94893","Type":"ContainerDied","Data":"c334d272a1ad477467e0ada227cd84adebefb119aae06169ad12fee419856934"}
Nov 03 20:26:31.803647 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.803614    5600 scope.go:122] "RemoveContainer" containerID="0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b"
Nov 03 20:26:31.803766 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.803709    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"
Nov 03 20:26:31.804986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.804954    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-tmp-dir\") pod \"427c352d-209c-43fd-9afd-0a0209d94893\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") "
Nov 03 20:26:31.805151 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.805001    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-model-cache\") pod \"427c352d-209c-43fd-9afd-0a0209d94893\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") "
Nov 03 20:26:31.805151 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.805041    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/427c352d-209c-43fd-9afd-0a0209d94893-tls-certs\" (UniqueName: \"kubernetes.io/secret/427c352d-209c-43fd-9afd-0a0209d94893-tls-certs\") pod \"427c352d-209c-43fd-9afd-0a0209d94893\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") "
Nov 03 20:26:31.805276 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.805123    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-home\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-home\") pod \"427c352d-209c-43fd-9afd-0a0209d94893\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") "
Nov 03 20:26:31.805276 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.805214    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-tmp-dir" pod "427c352d-209c-43fd-9afd-0a0209d94893" (UID: "427c352d-209c-43fd-9afd-0a0209d94893"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.805384 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.805322    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-dshm\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-dshm\") pod \"427c352d-209c-43fd-9afd-0a0209d94893\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") "
Nov 03 20:26:31.805563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.805381    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-model-cache" pod "427c352d-209c-43fd-9afd-0a0209d94893" (UID: "427c352d-209c-43fd-9afd-0a0209d94893"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.805563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.805420    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427c352d-209c-43fd-9afd-0a0209d94893-kube-api-access-nrsn4\" (UniqueName: \"kubernetes.io/projected/427c352d-209c-43fd-9afd-0a0209d94893-kube-api-access-nrsn4\") pod \"427c352d-209c-43fd-9afd-0a0209d94893\" (UID: \"427c352d-209c-43fd-9afd-0a0209d94893\") "
Nov 03 20:26:31.805767 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.805743    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-home" pod "427c352d-209c-43fd-9afd-0a0209d94893" (UID: "427c352d-209c-43fd-9afd-0a0209d94893"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.806050 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806028    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.806218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806053    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424be2c2-94d5-42d9-b434-d2bdaa809e85-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.806218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806068    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x6r6\" (UniqueName: \"kubernetes.io/projected/424be2c2-94d5-42d9-b434-d2bdaa809e85-kube-api-access-4x6r6\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.806218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806080    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424be2c2-94d5-42d9-b434-d2bdaa809e85-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.806218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806095    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdr76\" (UniqueName: \"kubernetes.io/projected/5d1e6564-0a1c-40d3-b800-665251906b46-kube-api-access-cdr76\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.806218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806107    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1e6564-0a1c-40d3-b800-665251906b46-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.806218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806121    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.806218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806132    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.806218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806152    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1e6564-0a1c-40d3-b800-665251906b46-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.806585 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806378    5600 generic.go:356] "Generic (PLEG): container finished" podID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerID="6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462" exitCode=0
Nov 03 20:26:31.806656 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806633    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" event={"ID":"424be2c2-94d5-42d9-b434-d2bdaa809e85","Type":"ContainerDied","Data":"6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462"}
Nov 03 20:26:31.806713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806674    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k" event={"ID":"424be2c2-94d5-42d9-b434-d2bdaa809e85","Type":"ContainerDied","Data":"2a0b1f07ad1c438435b849b0d515039d8e6649d756bc3ca132e5bbeb4af80ba5"}
Nov 03 20:26:31.806753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.806745    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"
Nov 03 20:26:31.822980 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.820176    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/427c352d-209c-43fd-9afd-0a0209d94893-tls-certs" pod "427c352d-209c-43fd-9afd-0a0209d94893" (UID: "427c352d-209c-43fd-9afd-0a0209d94893"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:31.822980 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.820270    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-dshm" pod "427c352d-209c-43fd-9afd-0a0209d94893" (UID: "427c352d-209c-43fd-9afd-0a0209d94893"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:31.833971 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.833883    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427c352d-209c-43fd-9afd-0a0209d94893-kube-api-access-nrsn4" pod "427c352d-209c-43fd-9afd-0a0209d94893" (UID: "427c352d-209c-43fd-9afd-0a0209d94893"). InnerVolumeSpecName "kube-api-access-nrsn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:31.839027 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.838990    5600 generic.go:356] "Generic (PLEG): container finished" podID="5d1e6564-0a1c-40d3-b800-665251906b46" containerID="0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef" exitCode=0
Nov 03 20:26:31.839178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.839058    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" event={"ID":"5d1e6564-0a1c-40d3-b800-665251906b46","Type":"ContainerDied","Data":"0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef"}
Nov 03 20:26:31.839178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.839097    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x" event={"ID":"5d1e6564-0a1c-40d3-b800-665251906b46","Type":"ContainerDied","Data":"cfde90ee0bb6b551321c189f5be6f6f0228b71d26b725c9a34f3ce00f7192c7d"}
Nov 03 20:26:31.839178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.839177    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"
Nov 03 20:26:31.840693 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.840672    5600 scope.go:122] "RemoveContainer" containerID="0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b"
Nov 03 20:26:31.841057 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:31.841037    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b\": container with ID starting with 0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b not found: ID does not exist" containerID="0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b"
Nov 03 20:26:31.841109 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.841067    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b"} err="failed to get container status \"0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b\": rpc error: code = NotFound desc = could not find container \"0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b\": container with ID starting with 0d9d4b71e01b7d46d95a6a7fa0b7ec50082cd730c3701e86857bc525a059627b not found: ID does not exist"
Nov 03 20:26:31.841109 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.841083    5600 scope.go:122] "RemoveContainer" containerID="6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462"
Nov 03 20:26:31.849986 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.849959    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"]
Nov 03 20:26:31.851931 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.851912    5600 scope.go:122] "RemoveContainer" containerID="6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462"
Nov 03 20:26:31.852213 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:31.852194    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462\": container with ID starting with 6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462 not found: ID does not exist" containerID="6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462"
Nov 03 20:26:31.852273 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.852220    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462"} err="failed to get container status \"6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462\": rpc error: code = NotFound desc = could not find container \"6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462\": container with ID starting with 6f34d5ce852093d510778a4ef5081c62a063acd51a98f34ec5b87539b9834462 not found: ID does not exist"
Nov 03 20:26:31.852273 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.852239    5600 scope.go:122] "RemoveContainer" containerID="0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef"
Nov 03 20:26:31.856807 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.856789    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-w6-0ed729-kserve-69c776cbb4-q687k"]
Nov 03 20:26:31.862394 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.862379    5600 scope.go:122] "RemoveContainer" containerID="0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef"
Nov 03 20:26:31.862705 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:31.862686    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef\": container with ID starting with 0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef not found: ID does not exist" containerID="0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef"
Nov 03 20:26:31.862748 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.862712    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef"} err="failed to get container status \"0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef\": rpc error: code = NotFound desc = could not find container \"0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef\": container with ID starting with 0fbac22da8cfa393211608e78e3d1a557d499cf496968dcc05d3f66888054bef not found: ID does not exist"
Nov 03 20:26:31.866819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.866795    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:31.866933 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.866845    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" podUID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:31.867632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.867610    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"]
Nov 03 20:26:31.871257 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.871236    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/distinct-2-w6-0ed729-kserve-79668fc9f5-dtt5x"]
Nov 03 20:26:31.916464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.910779    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/427c352d-209c-43fd-9afd-0a0209d94893-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.916464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.910829    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/427c352d-209c-43fd-9afd-0a0209d94893-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:31.916464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:31.910845    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrsn4\" (UniqueName: \"kubernetes.io/projected/427c352d-209c-43fd-9afd-0a0209d94893-kube-api-access-nrsn4\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:32.132862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:32.132827    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"]
Nov 03 20:26:32.135720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:32.135695    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-0ed729/premium-w6-0ed729-kserve-745488b97c-5zxql"]
Nov 03 20:26:32.649985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:32.649943    5600 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:32.650153 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:32.650008    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:33.617372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:33.617328    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" path="/var/lib/kubelet/pods/424be2c2-94d5-42d9-b434-d2bdaa809e85/volumes"
Nov 03 20:26:33.617968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:33.617949    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427c352d-209c-43fd-9afd-0a0209d94893" path="/var/lib/kubelet/pods/427c352d-209c-43fd-9afd-0a0209d94893/volumes"
Nov 03 20:26:33.618386 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:33.618370    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" path="/var/lib/kubelet/pods/5d1e6564-0a1c-40d3-b800-665251906b46/volumes"
Nov 03 20:26:33.667104 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:33.667069    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"]
Nov 03 20:26:34.773274 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:34.773231    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"]
Nov 03 20:26:34.774003 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:34.773551    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9" podUID="03f9474d-1418-4a81-a37c-b0eb9b6e145b" containerName="maas-api" containerID="cri-o://e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2" gracePeriod=30
Nov 03 20:26:35.213306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.213270    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:26:35.298719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.298686    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-metrics-certs\") pod \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") "
Nov 03 20:26:35.298930 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.298756    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f9474d-1418-4a81-a37c-b0eb9b6e145b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f9474d-1418-4a81-a37c-b0eb9b6e145b-openshift-service-ca\") pod \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") "
Nov 03 20:26:35.298930 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.298825    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f9474d-1418-4a81-a37c-b0eb9b6e145b-kube-api-access-q5thb\" (UniqueName: \"kubernetes.io/projected/03f9474d-1418-4a81-a37c-b0eb9b6e145b-kube-api-access-q5thb\") pod \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") "
Nov 03 20:26:35.299042 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.298990    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-maas-api-tls\") pod \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\" (UID: \"03f9474d-1418-4a81-a37c-b0eb9b6e145b\") "
Nov 03 20:26:35.299777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.299735    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f9474d-1418-4a81-a37c-b0eb9b6e145b-openshift-service-ca" pod "03f9474d-1418-4a81-a37c-b0eb9b6e145b" (UID: "03f9474d-1418-4a81-a37c-b0eb9b6e145b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:35.301625 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.301595    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-metrics-certs" pod "03f9474d-1418-4a81-a37c-b0eb9b6e145b" (UID: "03f9474d-1418-4a81-a37c-b0eb9b6e145b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:35.301725 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.301633    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-maas-api-tls" pod "03f9474d-1418-4a81-a37c-b0eb9b6e145b" (UID: "03f9474d-1418-4a81-a37c-b0eb9b6e145b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:35.301809 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.301783    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f9474d-1418-4a81-a37c-b0eb9b6e145b-kube-api-access-q5thb" pod "03f9474d-1418-4a81-a37c-b0eb9b6e145b" (UID: "03f9474d-1418-4a81-a37c-b0eb9b6e145b"). InnerVolumeSpecName "kube-api-access-q5thb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.400675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.400644    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.400768 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.400681    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f9474d-1418-4a81-a37c-b0eb9b6e145b-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.400768 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.400697    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5thb\" (UniqueName: \"kubernetes.io/projected/03f9474d-1418-4a81-a37c-b0eb9b6e145b-kube-api-access-q5thb\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.400768 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.400712    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f9474d-1418-4a81-a37c-b0eb9b6e145b-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.692554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.692479    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:26:35.744528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.744499    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:26:35.806341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.805992    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-token\" (UniqueName: \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-token\") pod \"139b0032-cd8f-426e-bdee-77ebb7955b50\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") "
Nov 03 20:26:35.806341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806049    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-socket\") pod \"139b0032-cd8f-426e-bdee-77ebb7955b50\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") "
Nov 03 20:26:35.806341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806092    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-credential-socket\") pod \"139b0032-cd8f-426e-bdee-77ebb7955b50\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") "
Nov 03 20:26:35.806341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806182    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istiod-ca-cert\") pod \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") "
Nov 03 20:26:35.806341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806215    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-envoy\") pod \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") "
Nov 03 20:26:35.806341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806258    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-data\") pod \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") "
Nov 03 20:26:35.806341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806296    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-kube-api-access-n4kkb\" (UniqueName: \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-kube-api-access-n4kkb\") pod \"139b0032-cd8f-426e-bdee-77ebb7955b50\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") "
Nov 03 20:26:35.806341 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806329    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-certs\") pod \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") "
Nov 03 20:26:35.807233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806393    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-podinfo\") pod \"139b0032-cd8f-426e-bdee-77ebb7955b50\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") "
Nov 03 20:26:35.807233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806471    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-podinfo\") pod \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") "
Nov 03 20:26:35.807233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806512    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-socket\") pod \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") "
Nov 03 20:26:35.807233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806567    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-token\" (UniqueName: \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-token\") pod \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") "
Nov 03 20:26:35.807233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806601    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-credential-socket\") pod \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") "
Nov 03 20:26:35.807233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806633    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-certs\") pod \"139b0032-cd8f-426e-bdee-77ebb7955b50\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") "
Nov 03 20:26:35.807233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806716    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/139b0032-cd8f-426e-bdee-77ebb7955b50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139b0032-cd8f-426e-bdee-77ebb7955b50-istiod-ca-cert\") pod \"139b0032-cd8f-426e-bdee-77ebb7955b50\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") "
Nov 03 20:26:35.807233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806753    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-envoy\") pod \"139b0032-cd8f-426e-bdee-77ebb7955b50\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") "
Nov 03 20:26:35.807233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806828    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-data\") pod \"139b0032-cd8f-426e-bdee-77ebb7955b50\" (UID: \"139b0032-cd8f-426e-bdee-77ebb7955b50\") "
Nov 03 20:26:35.807233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.806855    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-kube-api-access-cm7z9\" (UniqueName: \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-kube-api-access-cm7z9\") pod \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\" (UID: \"ba674abb-7d11-4a99-b3a1-e630b6d41a48\") "
Nov 03 20:26:35.808064 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.808031    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-credential-socket" pod "ba674abb-7d11-4a99-b3a1-e630b6d41a48" (UID: "ba674abb-7d11-4a99-b3a1-e630b6d41a48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.810302 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.808799    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-socket" pod "ba674abb-7d11-4a99-b3a1-e630b6d41a48" (UID: "ba674abb-7d11-4a99-b3a1-e630b6d41a48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.810302 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.809043    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-data" pod "ba674abb-7d11-4a99-b3a1-e630b6d41a48" (UID: "ba674abb-7d11-4a99-b3a1-e630b6d41a48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.810302 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.809066    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-certs" pod "139b0032-cd8f-426e-bdee-77ebb7955b50" (UID: "139b0032-cd8f-426e-bdee-77ebb7955b50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.810302 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.809249    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istiod-ca-cert" pod "ba674abb-7d11-4a99-b3a1-e630b6d41a48" (UID: "ba674abb-7d11-4a99-b3a1-e630b6d41a48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:35.810302 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.809656    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139b0032-cd8f-426e-bdee-77ebb7955b50-istiod-ca-cert" pod "139b0032-cd8f-426e-bdee-77ebb7955b50" (UID: "139b0032-cd8f-426e-bdee-77ebb7955b50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:35.810944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.810684    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-certs" pod "ba674abb-7d11-4a99-b3a1-e630b6d41a48" (UID: "ba674abb-7d11-4a99-b3a1-e630b6d41a48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.814615 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.814590    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-socket" pod "139b0032-cd8f-426e-bdee-77ebb7955b50" (UID: "139b0032-cd8f-426e-bdee-77ebb7955b50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.815128 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.814794    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-credential-socket" pod "139b0032-cd8f-426e-bdee-77ebb7955b50" (UID: "139b0032-cd8f-426e-bdee-77ebb7955b50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.815128 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.815012    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-data" pod "139b0032-cd8f-426e-bdee-77ebb7955b50" (UID: "139b0032-cd8f-426e-bdee-77ebb7955b50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.815283 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.815126    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-token" pod "139b0032-cd8f-426e-bdee-77ebb7955b50" (UID: "139b0032-cd8f-426e-bdee-77ebb7955b50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.815552 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.815424    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-kube-api-access-n4kkb" pod "139b0032-cd8f-426e-bdee-77ebb7955b50" (UID: "139b0032-cd8f-426e-bdee-77ebb7955b50"). InnerVolumeSpecName "kube-api-access-n4kkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.816944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.816908    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-envoy" pod "139b0032-cd8f-426e-bdee-77ebb7955b50" (UID: "139b0032-cd8f-426e-bdee-77ebb7955b50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.817385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.817347    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-podinfo" pod "139b0032-cd8f-426e-bdee-77ebb7955b50" (UID: "139b0032-cd8f-426e-bdee-77ebb7955b50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:35.819033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.818845    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-token" pod "ba674abb-7d11-4a99-b3a1-e630b6d41a48" (UID: "ba674abb-7d11-4a99-b3a1-e630b6d41a48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.819033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.818848    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-kube-api-access-cm7z9" pod "ba674abb-7d11-4a99-b3a1-e630b6d41a48" (UID: "ba674abb-7d11-4a99-b3a1-e630b6d41a48"). InnerVolumeSpecName "kube-api-access-cm7z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.819033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.818999    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-podinfo" pod "ba674abb-7d11-4a99-b3a1-e630b6d41a48" (UID: "ba674abb-7d11-4a99-b3a1-e630b6d41a48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:35.821276 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.821249    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-envoy" pod "ba674abb-7d11-4a99-b3a1-e630b6d41a48" (UID: "ba674abb-7d11-4a99-b3a1-e630b6d41a48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.826267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.826236    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"]
Nov 03 20:26:35.874139 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.874095    5600 generic.go:356] "Generic (PLEG): container finished" podID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerID="97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4" exitCode=0
Nov 03 20:26:35.874342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.874165    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"
Nov 03 20:26:35.874342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.874182    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" event={"ID":"139b0032-cd8f-426e-bdee-77ebb7955b50","Type":"ContainerDied","Data":"97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4"}
Nov 03 20:26:35.874342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.874275    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9" event={"ID":"139b0032-cd8f-426e-bdee-77ebb7955b50","Type":"ContainerDied","Data":"b3b640624b87ee729fd0ecc32345cbbee75ac648325686a6edfdfa2381fe7818"}
Nov 03 20:26:35.874342 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.874305    5600 scope.go:122] "RemoveContainer" containerID="97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4"
Nov 03 20:26:35.876393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.876077    5600 generic.go:356] "Generic (PLEG): container finished" podID="03f9474d-1418-4a81-a37c-b0eb9b6e145b" containerID="e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2" exitCode=0
Nov 03 20:26:35.876393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.876124    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9" event={"ID":"03f9474d-1418-4a81-a37c-b0eb9b6e145b","Type":"ContainerDied","Data":"e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2"}
Nov 03 20:26:35.876393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.876196    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9" event={"ID":"03f9474d-1418-4a81-a37c-b0eb9b6e145b","Type":"ContainerDied","Data":"c6e9aa53ddf7b2e272079dee2dab9ee775ffedc65f84b312999b18f7a6bf5085"}
Nov 03 20:26:35.876393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.876135    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"
Nov 03 20:26:35.878390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.878363    5600 generic.go:356] "Generic (PLEG): container finished" podID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" containerID="c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02" exitCode=0
Nov 03 20:26:35.878535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.878426    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" event={"ID":"ba674abb-7d11-4a99-b3a1-e630b6d41a48","Type":"ContainerDied","Data":"c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02"}
Nov 03 20:26:35.878535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.878487    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49" event={"ID":"ba674abb-7d11-4a99-b3a1-e630b6d41a48","Type":"ContainerDied","Data":"4f19ffe4ac8f85f0ef46da6cdcfaf1b07e14f803a89e7f69304f28ace5529583"}
Nov 03 20:26:35.878535 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.878490    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"
Nov 03 20:26:35.889412 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.889394    5600 scope.go:122] "RemoveContainer" containerID="97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4"
Nov 03 20:26:35.889785 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:35.889764    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4\": container with ID starting with 97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4 not found: ID does not exist" containerID="97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4"
Nov 03 20:26:35.889853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.889795    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4"} err="failed to get container status \"97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4\": rpc error: code = NotFound desc = could not find container \"97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4\": container with ID starting with 97d500e4c9982c3c682b83675587519ef9f5b30d671b7761854cbe00055574e4 not found: ID does not exist"
Nov 03 20:26:35.889853 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.889813    5600 scope.go:122] "RemoveContainer" containerID="e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2"
Nov 03 20:26:35.903823 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.903698    5600 scope.go:122] "RemoveContainer" containerID="e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2"
Nov 03 20:26:35.904500 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:35.904468    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2\": container with ID starting with e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2 not found: ID does not exist" containerID="e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2"
Nov 03 20:26:35.904632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.904508    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2"} err="failed to get container status \"e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2\": rpc error: code = NotFound desc = could not find container \"e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2\": container with ID starting with e5e1c8bbea1a99b399527db391ec346098c59dbae570b22978b2fea42eb8abc2 not found: ID does not exist"
Nov 03 20:26:35.904632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.904534    5600 scope.go:122] "RemoveContainer" containerID="c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02"
Nov 03 20:26:35.904860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.904839    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"]
Nov 03 20:26:35.907581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.907553    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-27d3ad-d9d68b96-8cgn9"]
Nov 03 20:26:35.908307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908280    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908313    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908329    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908342    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908355    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908369    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908382    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/139b0032-cd8f-426e-bdee-77ebb7955b50-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908395    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908408    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908421    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm7z9\" (UniqueName: \"kubernetes.io/projected/ba674abb-7d11-4a99-b3a1-e630b6d41a48-kube-api-access-cm7z9\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908454    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908468    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908481    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/139b0032-cd8f-426e-bdee-77ebb7955b50-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908496    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908508    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908523    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908537    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4kkb\" (UniqueName: \"kubernetes.io/projected/139b0032-cd8f-426e-bdee-77ebb7955b50-kube-api-access-n4kkb\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.908938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.908552    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba674abb-7d11-4a99-b3a1-e630b6d41a48-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.916797 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.916773    5600 scope.go:122] "RemoveContainer" containerID="c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02"
Nov 03 20:26:35.917166 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:35.917144    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02\": container with ID starting with c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02 not found: ID does not exist" containerID="c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02"
Nov 03 20:26:35.917233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.917175    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02"} err="failed to get container status \"c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02\": rpc error: code = NotFound desc = could not find container \"c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02\": container with ID starting with c78fbdd422ef7d0f4e9665cce709896ee38f21f8f72d629d5bbab3b0e7df6b02 not found: ID does not exist"
Nov 03 20:26:35.926233 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.926196    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"]
Nov 03 20:26:35.928729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.928709    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-28hk9"]
Nov 03 20:26:35.936888 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.936857    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"]
Nov 03 20:26:35.940516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:35.940492    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-j4q49"]
Nov 03 20:26:36.001919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.001818    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"]
Nov 03 20:26:36.008069 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.008030    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv" podUID="e31ccaed-b4ee-416a-b88c-3ceefecd645a" containerName="maas-api" containerID="cri-o://44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50" gracePeriod=30
Nov 03 20:26:36.166846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.165860    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"]
Nov 03 20:26:36.166846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.166119    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs" podUID="6d428d2f-5d0d-4fab-94d8-5efe40e8e49a" containerName="maas-api" containerID="cri-o://d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a" gracePeriod=30
Nov 03 20:26:36.473778 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.473488    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:26:36.537210 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.536965    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-metrics-certs\") pod \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") "
Nov 03 20:26:36.537210 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.537124    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31ccaed-b4ee-416a-b88c-3ceefecd645a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31ccaed-b4ee-416a-b88c-3ceefecd645a-openshift-service-ca\") pod \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") "
Nov 03 20:26:36.537210 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.537208    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31ccaed-b4ee-416a-b88c-3ceefecd645a-kube-api-access-8tmzf\" (UniqueName: \"kubernetes.io/projected/e31ccaed-b4ee-416a-b88c-3ceefecd645a-kube-api-access-8tmzf\") pod \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") "
Nov 03 20:26:36.537508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.537260    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-maas-api-tls\") pod \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\" (UID: \"e31ccaed-b4ee-416a-b88c-3ceefecd645a\") "
Nov 03 20:26:36.543213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.542049    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-maas-api-tls" pod "e31ccaed-b4ee-416a-b88c-3ceefecd645a" (UID: "e31ccaed-b4ee-416a-b88c-3ceefecd645a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:36.543599 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.543539    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31ccaed-b4ee-416a-b88c-3ceefecd645a-openshift-service-ca" pod "e31ccaed-b4ee-416a-b88c-3ceefecd645a" (UID: "e31ccaed-b4ee-416a-b88c-3ceefecd645a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:36.555463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.555379    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-metrics-certs" pod "e31ccaed-b4ee-416a-b88c-3ceefecd645a" (UID: "e31ccaed-b4ee-416a-b88c-3ceefecd645a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:36.560624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.560547    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31ccaed-b4ee-416a-b88c-3ceefecd645a-kube-api-access-8tmzf" pod "e31ccaed-b4ee-416a-b88c-3ceefecd645a" (UID: "e31ccaed-b4ee-416a-b88c-3ceefecd645a"). InnerVolumeSpecName "kube-api-access-8tmzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:36.631091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.631060    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:26:36.638354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.638321    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-maas-api-tls\") pod \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") "
Nov 03 20:26:36.638543 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.638454    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-metrics-certs\") pod \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") "
Nov 03 20:26:36.638543 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.638507    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-openshift-service-ca\") pod \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") "
Nov 03 20:26:36.638632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.638551    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-kube-api-access-6dlzn\" (UniqueName: \"kubernetes.io/projected/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-kube-api-access-6dlzn\") pod \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\" (UID: \"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a\") "
Nov 03 20:26:36.638878 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.638851    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-openshift-service-ca" pod "6d428d2f-5d0d-4fab-94d8-5efe40e8e49a" (UID: "6d428d2f-5d0d-4fab-94d8-5efe40e8e49a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:36.639101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.639078    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31ccaed-b4ee-416a-b88c-3ceefecd645a-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:36.639166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.639107    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:36.639166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.639122    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tmzf\" (UniqueName: \"kubernetes.io/projected/e31ccaed-b4ee-416a-b88c-3ceefecd645a-kube-api-access-8tmzf\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:36.639166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.639138    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:36.639166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.639152    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e31ccaed-b4ee-416a-b88c-3ceefecd645a-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:36.640824 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.640797    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-metrics-certs" pod "6d428d2f-5d0d-4fab-94d8-5efe40e8e49a" (UID: "6d428d2f-5d0d-4fab-94d8-5efe40e8e49a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:36.640967 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.640947    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-maas-api-tls" pod "6d428d2f-5d0d-4fab-94d8-5efe40e8e49a" (UID: "6d428d2f-5d0d-4fab-94d8-5efe40e8e49a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:36.641643 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.641622    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-kube-api-access-6dlzn" pod "6d428d2f-5d0d-4fab-94d8-5efe40e8e49a" (UID: "6d428d2f-5d0d-4fab-94d8-5efe40e8e49a"). InnerVolumeSpecName "kube-api-access-6dlzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:36.740136 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.740095    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:36.740136 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.740129    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:36.740136 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.740139    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dlzn\" (UniqueName: \"kubernetes.io/projected/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a-kube-api-access-6dlzn\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:36.888935 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.888905    5600 generic.go:356] "Generic (PLEG): container finished" podID="e31ccaed-b4ee-416a-b88c-3ceefecd645a" containerID="44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50" exitCode=0
Nov 03 20:26:36.890482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.888976    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"
Nov 03 20:26:36.890482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.888994    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv" event={"ID":"e31ccaed-b4ee-416a-b88c-3ceefecd645a","Type":"ContainerDied","Data":"44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50"}
Nov 03 20:26:36.890482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.889602    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv" event={"ID":"e31ccaed-b4ee-416a-b88c-3ceefecd645a","Type":"ContainerDied","Data":"7130619ee05e04978d1d5fe0e485077082500d83fb1e2ee484ddb35be0cefadc"}
Nov 03 20:26:36.890482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.889651    5600 scope.go:122] "RemoveContainer" containerID="44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50"
Nov 03 20:26:36.893238 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.893205    5600 generic.go:356] "Generic (PLEG): container finished" podID="6d428d2f-5d0d-4fab-94d8-5efe40e8e49a" containerID="d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a" exitCode=0
Nov 03 20:26:36.893352 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.893330    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs" event={"ID":"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a","Type":"ContainerDied","Data":"d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a"}
Nov 03 20:26:36.893424 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.893368    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs" event={"ID":"6d428d2f-5d0d-4fab-94d8-5efe40e8e49a","Type":"ContainerDied","Data":"db8a4598f3cf0e79831cc1185505260697cda4faf336102e970265a7ab9a44be"}
Nov 03 20:26:36.893424 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.893421    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"
Nov 03 20:26:36.908950 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.908924    5600 scope.go:122] "RemoveContainer" containerID="44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50"
Nov 03 20:26:36.909315 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:36.909291    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50\": container with ID starting with 44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50 not found: ID does not exist" containerID="44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50"
Nov 03 20:26:36.909458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.909323    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50"} err="failed to get container status \"44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50\": rpc error: code = NotFound desc = could not find container \"44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50\": container with ID starting with 44fc3b7a2984505f0e053bbbddba3641b0a4c35ffd2dee019ff3a6fdc445dc50 not found: ID does not exist"
Nov 03 20:26:36.909458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.909343    5600 scope.go:122] "RemoveContainer" containerID="d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a"
Nov 03 20:26:36.920464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.920290    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"]
Nov 03 20:26:36.922045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.922027    5600 scope.go:122] "RemoveContainer" containerID="d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a"
Nov 03 20:26:36.923106 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:36.923076    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a\": container with ID starting with d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a not found: ID does not exist" containerID="d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a"
Nov 03 20:26:36.923268 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.923115    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a"} err="failed to get container status \"d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a\": rpc error: code = NotFound desc = could not find container \"d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a\": container with ID starting with d14f6451316bff7df13a235d96345b9c454a90c0358cc1dfab5a02b2ed4dd95a not found: ID does not exist"
Nov 03 20:26:36.926990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.926963    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-944e83-669b854db7-l2gpv"]
Nov 03 20:26:36.937002 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.936978    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"]
Nov 03 20:26:36.940932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:36.940908    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0ed729-7f7bfb48d4-wjsfs"]
Nov 03 20:26:37.618320 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:37.618285    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f9474d-1418-4a81-a37c-b0eb9b6e145b" path="/var/lib/kubelet/pods/03f9474d-1418-4a81-a37c-b0eb9b6e145b/volumes"
Nov 03 20:26:37.618924 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:37.618900    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" path="/var/lib/kubelet/pods/139b0032-cd8f-426e-bdee-77ebb7955b50/volumes"
Nov 03 20:26:37.619299 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:37.619282    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d428d2f-5d0d-4fab-94d8-5efe40e8e49a" path="/var/lib/kubelet/pods/6d428d2f-5d0d-4fab-94d8-5efe40e8e49a/volumes"
Nov 03 20:26:37.620079 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:37.620058    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" path="/var/lib/kubelet/pods/ba674abb-7d11-4a99-b3a1-e630b6d41a48/volumes"
Nov 03 20:26:37.620474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:37.620458    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31ccaed-b4ee-416a-b88c-3ceefecd645a" path="/var/lib/kubelet/pods/e31ccaed-b4ee-416a-b88c-3ceefecd645a/volumes"
Nov 03 20:26:45.097112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:45.097070    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"]
Nov 03 20:26:45.115899 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:45.114489    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy" containerID="cri-o://5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913" gracePeriod=30
Nov 03 20:26:47.354645 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.354605    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"]
Nov 03 20:26:47.355181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.354863    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz" podUID="b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd" containerName="maas-api" containerID="cri-o://d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8" gracePeriod=30
Nov 03 20:26:47.754228 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.754192    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:26:47.805619 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.805585    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-metrics-certs\") pod \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") "
Nov 03 20:26:47.805816 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.805628    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-openshift-service-ca\") pod \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") "
Nov 03 20:26:47.805816 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.805679    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-kube-api-access-mhcjz\" (UniqueName: \"kubernetes.io/projected/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-kube-api-access-mhcjz\") pod \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") "
Nov 03 20:26:47.805816 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.805749    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-maas-api-tls\") pod \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\" (UID: \"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd\") "
Nov 03 20:26:47.806378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.806057    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-openshift-service-ca" pod "b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd" (UID: "b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:47.806378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.806209    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.808932 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.808895    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-maas-api-tls" pod "b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd" (UID: "b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:47.809113 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.809091    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-metrics-certs" pod "b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd" (UID: "b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:47.810695 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.809617    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-kube-api-access-mhcjz" pod "b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd" (UID: "b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd"). InnerVolumeSpecName "kube-api-access-mhcjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:47.907917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.907820    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhcjz\" (UniqueName: \"kubernetes.io/projected/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-kube-api-access-mhcjz\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.907917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.907858    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.907917 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.907868    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.994819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.994709    5600 generic.go:356] "Generic (PLEG): container finished" podID="b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd" containerID="d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8" exitCode=0
Nov 03 20:26:47.994819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.994767    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"
Nov 03 20:26:47.994819 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.994779    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz" event={"ID":"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd","Type":"ContainerDied","Data":"d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8"}
Nov 03 20:26:47.995127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.994825    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz" event={"ID":"b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd","Type":"ContainerDied","Data":"6562f2b518e2aec4c40f8da4f9938128697cbbfd07d937e0776ac7194a3adc5e"}
Nov 03 20:26:47.995127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.994849    5600 scope.go:122] "RemoveContainer" containerID="d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8"
Nov 03 20:26:47.995565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:47.995302    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"]
Nov 03 20:26:48.009066 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:48.009043    5600 scope.go:122] "RemoveContainer" containerID="d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8"
Nov 03 20:26:48.009487 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:48.009415    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8\": container with ID starting with d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8 not found: ID does not exist" containerID="d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8"
Nov 03 20:26:48.009783 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:48.009499    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8"} err="failed to get container status \"d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8\": rpc error: code = NotFound desc = could not find container \"d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8\": container with ID starting with d7e487807c94051e5fe36245be0a56c48d8d403b415aacb036b0cf1f9cde8bb8 not found: ID does not exist"
Nov 03 20:26:48.046478 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:48.046442    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"]
Nov 03 20:26:48.056349 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:48.056315    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-456f55-5bcc6c44c9-5t9bz"]
Nov 03 20:26:48.723320 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:48.723254    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm" podUID="fb286a49-946b-4a5f-a2c8-6d116a51eb0b" containerName="main" containerID="cri-o://4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654" gracePeriod=45
Nov 03 20:26:48.984176 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:48.984111    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:26:49.006500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.006462    5600 generic.go:356] "Generic (PLEG): container finished" podID="fb286a49-946b-4a5f-a2c8-6d116a51eb0b" containerID="4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654" exitCode=0
Nov 03 20:26:49.006672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.006515    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm" event={"ID":"fb286a49-946b-4a5f-a2c8-6d116a51eb0b","Type":"ContainerDied","Data":"4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654"}
Nov 03 20:26:49.006672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.006544    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm" event={"ID":"fb286a49-946b-4a5f-a2c8-6d116a51eb0b","Type":"ContainerDied","Data":"6fb6bad49dcbae401580302069b5afd3c33166aec0a34bff8122b7f8acdefe41"}
Nov 03 20:26:49.006672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.006565    5600 scope.go:122] "RemoveContainer" containerID="4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654"
Nov 03 20:26:49.006672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.006655    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"
Nov 03 20:26:49.022020 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.021933    5600 scope.go:122] "RemoveContainer" containerID="4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654"
Nov 03 20:26:49.022381 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:49.022350    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654\": container with ID starting with 4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654 not found: ID does not exist" containerID="4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654"
Nov 03 20:26:49.022554 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.022393    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654"} err="failed to get container status \"4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654\": rpc error: code = NotFound desc = could not find container \"4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654\": container with ID starting with 4a8e1837d288b0e5eef0bb9326309b2d41cace84ec1c7fb1c51637a69529b654 not found: ID does not exist"
Nov 03 20:26:49.123523 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.123490    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-dshm\") pod \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") "
Nov 03 20:26:49.123715 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.123556    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-home\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-home\") pod \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") "
Nov 03 20:26:49.123715 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.123588    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tls-certs\") pod \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") "
Nov 03 20:26:49.123715 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.123659    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-model-cache\") pod \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") "
Nov 03 20:26:49.124224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.123734    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-kube-api-access-lq59m\" (UniqueName: \"kubernetes.io/projected/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-kube-api-access-lq59m\") pod \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") "
Nov 03 20:26:49.124224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.123789    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tmp-dir\") pod \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\" (UID: \"fb286a49-946b-4a5f-a2c8-6d116a51eb0b\") "
Nov 03 20:26:49.124224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.123835    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-home" pod "fb286a49-946b-4a5f-a2c8-6d116a51eb0b" (UID: "fb286a49-946b-4a5f-a2c8-6d116a51eb0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:49.124224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.123973    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-model-cache" pod "fb286a49-946b-4a5f-a2c8-6d116a51eb0b" (UID: "fb286a49-946b-4a5f-a2c8-6d116a51eb0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:49.124224 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.124087    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tmp-dir" pod "fb286a49-946b-4a5f-a2c8-6d116a51eb0b" (UID: "fb286a49-946b-4a5f-a2c8-6d116a51eb0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:49.124559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.124348    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:49.124559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.124371    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:49.124559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.124387    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:49.126185 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.126159    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-kube-api-access-lq59m" pod "fb286a49-946b-4a5f-a2c8-6d116a51eb0b" (UID: "fb286a49-946b-4a5f-a2c8-6d116a51eb0b"). InnerVolumeSpecName "kube-api-access-lq59m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:49.126549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.126531    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-dshm" pod "fb286a49-946b-4a5f-a2c8-6d116a51eb0b" (UID: "fb286a49-946b-4a5f-a2c8-6d116a51eb0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:49.126612 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.126551    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tls-certs" pod "fb286a49-946b-4a5f-a2c8-6d116a51eb0b" (UID: "fb286a49-946b-4a5f-a2c8-6d116a51eb0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:49.225997 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.225955    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq59m\" (UniqueName: \"kubernetes.io/projected/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-kube-api-access-lq59m\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:49.225997 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.225990    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:49.225997 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.226002    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb286a49-946b-4a5f-a2c8-6d116a51eb0b-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:49.328684 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.328656    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"]
Nov 03 20:26:49.332925 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.332898    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b55105/auth-test-model-b55105-kserve-55f544647c-qxlbm"]
Nov 03 20:26:49.618228 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.618136    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd" path="/var/lib/kubelet/pods/b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd/volumes"
Nov 03 20:26:49.618675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:49.618657    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb286a49-946b-4a5f-a2c8-6d116a51eb0b" path="/var/lib/kubelet/pods/fb286a49-946b-4a5f-a2c8-6d116a51eb0b/volumes"
Nov 03 20:26:50.410053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.409643    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:26:50.542811 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.542773    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-credential-socket\") pod \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") "
Nov 03 20:26:50.543036 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.542863    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-socket\") pod \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") "
Nov 03 20:26:50.543036 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.542910    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-kube-api-access-xs7cx\" (UniqueName: \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-kube-api-access-xs7cx\") pod \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") "
Nov 03 20:26:50.543036 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.542970    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istiod-ca-cert\") pod \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") "
Nov 03 20:26:50.543036 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.543007    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-token\" (UniqueName: \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-token\") pod \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") "
Nov 03 20:26:50.543267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.543041    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-envoy\") pod \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") "
Nov 03 20:26:50.543267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.543148    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-data\") pod \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") "
Nov 03 20:26:50.543267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.543160    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-credential-socket" pod "f2d84a43-032e-4b91-a73e-16f6d083fbf7" (UID: "f2d84a43-032e-4b91-a73e-16f6d083fbf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:50.543267 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.543202    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-podinfo\") pod \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") "
Nov 03 20:26:50.543531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.543481    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istiod-ca-cert" pod "f2d84a43-032e-4b91-a73e-16f6d083fbf7" (UID: "f2d84a43-032e-4b91-a73e-16f6d083fbf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:50.543642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.543586    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-data" pod "f2d84a43-032e-4b91-a73e-16f6d083fbf7" (UID: "f2d84a43-032e-4b91-a73e-16f6d083fbf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:50.543829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.543244    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-certs\") pod \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\" (UID: \"f2d84a43-032e-4b91-a73e-16f6d083fbf7\") "
Nov 03 20:26:50.543829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.543696    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-socket" pod "f2d84a43-032e-4b91-a73e-16f6d083fbf7" (UID: "f2d84a43-032e-4b91-a73e-16f6d083fbf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:50.543829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.543747    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-certs" pod "f2d84a43-032e-4b91-a73e-16f6d083fbf7" (UID: "f2d84a43-032e-4b91-a73e-16f6d083fbf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:50.544167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.544138    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.544286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.544175    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.544286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.544192    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.544286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.544205    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.544286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.544219    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.545564 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.545538    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-envoy" pod "f2d84a43-032e-4b91-a73e-16f6d083fbf7" (UID: "f2d84a43-032e-4b91-a73e-16f6d083fbf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:50.545719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.545700    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-kube-api-access-xs7cx" pod "f2d84a43-032e-4b91-a73e-16f6d083fbf7" (UID: "f2d84a43-032e-4b91-a73e-16f6d083fbf7"). InnerVolumeSpecName "kube-api-access-xs7cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:50.545887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.545868    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-podinfo" pod "f2d84a43-032e-4b91-a73e-16f6d083fbf7" (UID: "f2d84a43-032e-4b91-a73e-16f6d083fbf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:50.546011 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.545993    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-token" pod "f2d84a43-032e-4b91-a73e-16f6d083fbf7" (UID: "f2d84a43-032e-4b91-a73e-16f6d083fbf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:50.645553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.645514    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.645553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.645547    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.645553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.645556    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d84a43-032e-4b91-a73e-16f6d083fbf7-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.645862 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.645565    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs7cx\" (UniqueName: \"kubernetes.io/projected/f2d84a43-032e-4b91-a73e-16f6d083fbf7-kube-api-access-xs7cx\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.854138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:50.854034    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" containerID="cri-o://df22a4b39d0e96783baf1030381b0ac277bc4418e7a2ce96d65d05a7b04b6d54" gracePeriod=45
Nov 03 20:26:51.022387 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.022341    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused"
Nov 03 20:26:51.023965 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.023886    5600 generic.go:356] "Generic (PLEG): container finished" podID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerID="5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913" exitCode=0
Nov 03 20:26:51.024458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.024100    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" event={"ID":"f2d84a43-032e-4b91-a73e-16f6d083fbf7","Type":"ContainerDied","Data":"5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913"}
Nov 03 20:26:51.024458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.024153    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs" event={"ID":"f2d84a43-032e-4b91-a73e-16f6d083fbf7","Type":"ContainerDied","Data":"0d15638201f88769fb16e95ac65211224d2ea6251db9ff1cd2f0d58c5ca32b6d"}
Nov 03 20:26:51.024458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.024149    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"
Nov 03 20:26:51.024458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.024176    5600 scope.go:122] "RemoveContainer" containerID="5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913"
Nov 03 20:26:51.030882 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.030852    5600 generic.go:356] "Generic (PLEG): container finished" podID="447f04b7-254a-4df8-9581-db5e98d4b144" containerID="df22a4b39d0e96783baf1030381b0ac277bc4418e7a2ce96d65d05a7b04b6d54" exitCode=0
Nov 03 20:26:51.031012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.030919    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" event={"ID":"447f04b7-254a-4df8-9581-db5e98d4b144","Type":"ContainerDied","Data":"df22a4b39d0e96783baf1030381b0ac277bc4418e7a2ce96d65d05a7b04b6d54"}
Nov 03 20:26:51.049173 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.049120    5600 scope.go:122] "RemoveContainer" containerID="5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913"
Nov 03 20:26:51.049560 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:51.049537    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913\": container with ID starting with 5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913 not found: ID does not exist" containerID="5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913"
Nov 03 20:26:51.049656 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.049570    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913"} err="failed to get container status \"5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913\": rpc error: code = NotFound desc = could not find container \"5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913\": container with ID starting with 5b9c821fe73162545aea75bbc6ce64cab612b9d3c6794bec36ae65dc9180a913 not found: ID does not exist"
Nov 03 20:26:51.056939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.056914    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"]
Nov 03 20:26:51.082742 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.067840    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-52rzs"]
Nov 03 20:26:51.138729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.138700    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:26:51.251204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.251165    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-tmp-dir\") pod \"447f04b7-254a-4df8-9581-db5e98d4b144\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") "
Nov 03 20:26:51.251403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.251241    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-model-cache\") pod \"447f04b7-254a-4df8-9581-db5e98d4b144\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") "
Nov 03 20:26:51.251403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.251315    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447f04b7-254a-4df8-9581-db5e98d4b144-kube-api-access-qpv7f\" (UniqueName: \"kubernetes.io/projected/447f04b7-254a-4df8-9581-db5e98d4b144-kube-api-access-qpv7f\") pod \"447f04b7-254a-4df8-9581-db5e98d4b144\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") "
Nov 03 20:26:51.251403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.251358    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-home\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-home\") pod \"447f04b7-254a-4df8-9581-db5e98d4b144\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") "
Nov 03 20:26:51.251630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.251549    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-dshm\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-dshm\") pod \"447f04b7-254a-4df8-9581-db5e98d4b144\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") "
Nov 03 20:26:51.251630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.251552    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-tmp-dir" pod "447f04b7-254a-4df8-9581-db5e98d4b144" (UID: "447f04b7-254a-4df8-9581-db5e98d4b144"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.251630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.251567    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-model-cache" pod "447f04b7-254a-4df8-9581-db5e98d4b144" (UID: "447f04b7-254a-4df8-9581-db5e98d4b144"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.251790 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.251668    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/447f04b7-254a-4df8-9581-db5e98d4b144-tls-certs\" (UniqueName: \"kubernetes.io/secret/447f04b7-254a-4df8-9581-db5e98d4b144-tls-certs\") pod \"447f04b7-254a-4df8-9581-db5e98d4b144\" (UID: \"447f04b7-254a-4df8-9581-db5e98d4b144\") "
Nov 03 20:26:51.251849 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.251823    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-home" pod "447f04b7-254a-4df8-9581-db5e98d4b144" (UID: "447f04b7-254a-4df8-9581-db5e98d4b144"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.252162 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.252137    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.252283 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.252168    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.252283 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.252183    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.254597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.254569    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-dshm" pod "447f04b7-254a-4df8-9581-db5e98d4b144" (UID: "447f04b7-254a-4df8-9581-db5e98d4b144"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.254597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.254589    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/447f04b7-254a-4df8-9581-db5e98d4b144-tls-certs" pod "447f04b7-254a-4df8-9581-db5e98d4b144" (UID: "447f04b7-254a-4df8-9581-db5e98d4b144"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:51.255111 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.255088    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447f04b7-254a-4df8-9581-db5e98d4b144-kube-api-access-qpv7f" pod "447f04b7-254a-4df8-9581-db5e98d4b144" (UID: "447f04b7-254a-4df8-9581-db5e98d4b144"). InnerVolumeSpecName "kube-api-access-qpv7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:51.353278 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.353235    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/447f04b7-254a-4df8-9581-db5e98d4b144-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.353278 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.353280    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/447f04b7-254a-4df8-9581-db5e98d4b144-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.353524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.353296    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpv7f\" (UniqueName: \"kubernetes.io/projected/447f04b7-254a-4df8-9581-db5e98d4b144-kube-api-access-qpv7f\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.615913 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:51.615880    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" path="/var/lib/kubelet/pods/f2d84a43-032e-4b91-a73e-16f6d083fbf7/volumes"
Nov 03 20:26:52.038228 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:52.038198    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"
Nov 03 20:26:52.038228 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:52.038223    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg" event={"ID":"447f04b7-254a-4df8-9581-db5e98d4b144","Type":"ContainerDied","Data":"5abeb91466409bd8cd1509882a9a5afdd632d2819c1932072959186ec409358e"}
Nov 03 20:26:52.038575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:52.038268    5600 scope.go:122] "RemoveContainer" containerID="df22a4b39d0e96783baf1030381b0ac277bc4418e7a2ce96d65d05a7b04b6d54"
Nov 03 20:26:52.056606 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:52.056579    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"]
Nov 03 20:26:52.060269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:52.060241    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/distinct-2-w0-120fe3-kserve-66bdf77495-4hqxg"]
Nov 03 20:26:53.616126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:53.616092    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" path="/var/lib/kubelet/pods/447f04b7-254a-4df8-9581-db5e98d4b144/volumes"
Nov 03 20:26:55.106126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.106088    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"]
Nov 03 20:26:55.943620 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.943584    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"]
Nov 03 20:26:55.944164 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944145    5600 cpu_manager.go:414] "containerMap: removing container" podUID="e0121c77-f76c-4041-8dc3-707f115f5bcb" containerName="maas-api"
Nov 03 20:26:55.944244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944167    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0121c77-f76c-4041-8dc3-707f115f5bcb" containerName="maas-api"
Nov 03 20:26:55.944244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944198    5600 cpu_manager.go:414] "containerMap: removing container" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerName="istio-proxy"
Nov 03 20:26:55.944244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944205    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerName="istio-proxy"
Nov 03 20:26:55.944244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944215    5600 cpu_manager.go:414] "containerMap: removing container" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerName="istio-proxy"
Nov 03 20:26:55.944244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944222    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerName="istio-proxy"
Nov 03 20:26:55.944244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944234    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" containerName="istio-proxy"
Nov 03 20:26:55.944244 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944242    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" containerName="istio-proxy"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944252    5600 cpu_manager.go:414] "containerMap: removing container" podUID="b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd" containerName="maas-api"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944261    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd" containerName="maas-api"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944280    5600 cpu_manager.go:414] "containerMap: removing container" podUID="6d428d2f-5d0d-4fab-94d8-5efe40e8e49a" containerName="maas-api"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944286    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d428d2f-5d0d-4fab-94d8-5efe40e8e49a" containerName="maas-api"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944303    5600 cpu_manager.go:414] "containerMap: removing container" podUID="03f9474d-1418-4a81-a37c-b0eb9b6e145b" containerName="maas-api"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944311    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f9474d-1418-4a81-a37c-b0eb9b6e145b" containerName="maas-api"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944321    5600 cpu_manager.go:414] "containerMap: removing container" podUID="e31ccaed-b4ee-416a-b88c-3ceefecd645a" containerName="maas-api"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944327    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31ccaed-b4ee-416a-b88c-3ceefecd645a" containerName="maas-api"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944343    5600 cpu_manager.go:414] "containerMap: removing container" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944350    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944358    5600 cpu_manager.go:414] "containerMap: removing container" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944364    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944381    5600 cpu_manager.go:414] "containerMap: removing container" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944388    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944400    5600 cpu_manager.go:414] "containerMap: removing container" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944406    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944418    5600 cpu_manager.go:414] "containerMap: removing container" podUID="fb286a49-946b-4a5f-a2c8-6d116a51eb0b" containerName="main"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944424    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb286a49-946b-4a5f-a2c8-6d116a51eb0b" containerName="main"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944454    5600 cpu_manager.go:414] "containerMap: removing container" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main"
Nov 03 20:26:55.944512 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944462    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944567    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="447f04b7-254a-4df8-9581-db5e98d4b144" containerName="main"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944580    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d84a43-032e-4b91-a73e-16f6d083fbf7" containerName="istio-proxy"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944588    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="427c352d-209c-43fd-9afd-0a0209d94893" containerName="main"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944596    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1e6564-0a1c-40d3-b800-665251906b46" containerName="main"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944604    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb286a49-946b-4a5f-a2c8-6d116a51eb0b" containerName="main"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944612    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="424be2c2-94d5-42d9-b434-d2bdaa809e85" containerName="main"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944620    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0121c77-f76c-4041-8dc3-707f115f5bcb" containerName="maas-api"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944631    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="139b0032-cd8f-426e-bdee-77ebb7955b50" containerName="istio-proxy"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944641    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="5789c64e-f731-4af7-96c0-f044fd64aaf6" containerName="istio-proxy"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944648    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba674abb-7d11-4a99-b3a1-e630b6d41a48" containerName="istio-proxy"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944657    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1df3f04-e3c8-4aae-b58c-e8a9ab2bb2fd" containerName="maas-api"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944666    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d428d2f-5d0d-4fab-94d8-5efe40e8e49a" containerName="maas-api"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944675    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f9474d-1418-4a81-a37c-b0eb9b6e145b" containerName="maas-api"
Nov 03 20:26:55.945178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.944682    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31ccaed-b4ee-416a-b88c-3ceefecd645a" containerName="maas-api"
Nov 03 20:26:55.949854 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.949831    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:55.955809 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:55.955785    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"]
Nov 03 20:26:56.004175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.004131    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:26:56.004390 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.004370    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:26:56.122522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.122491    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:56.122937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.122536    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpp2g\" (UniqueName: \"kubernetes.io/projected/0c16de55-ec16-4a5d-bb99-8793dd848e5b-kube-api-access-dpp2g\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:56.122937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.122761    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:56.224258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.224164    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:56.224473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.224290    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:56.224473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.224325    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpp2g\" (UniqueName: \"kubernetes.io/projected/0c16de55-ec16-4a5d-bb99-8793dd848e5b-kube-api-access-dpp2g\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:56.224934 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.224903    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:56.226924 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.226901    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:56.232158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.232136    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpp2g\" (UniqueName: \"kubernetes.io/projected/0c16de55-ec16-4a5d-bb99-8793dd848e5b-kube-api-access-dpp2g\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:56.268378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.268348    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:56.550171 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:56.550134    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"]
Nov 03 20:26:56.551818 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:26:56.551788    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c16de55_ec16_4a5d_bb99_8793dd848e5b.slice/crio-fe202c225b90573f781a4fd23d982c7861f31b5cf97fbc5b6d55499f3207c8eb WatchSource:0}: Error finding container fe202c225b90573f781a4fd23d982c7861f31b5cf97fbc5b6d55499f3207c8eb: Status 404 returned error can't find the container with id fe202c225b90573f781a4fd23d982c7861f31b5cf97fbc5b6d55499f3207c8eb
Nov 03 20:26:57.051175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.051142    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"]
Nov 03 20:26:57.055144 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.055105    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:26:57.063581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.062982    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"]
Nov 03 20:26:57.077828 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.077795    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6" event={"ID":"0c16de55-ec16-4a5d-bb99-8793dd848e5b","Type":"ContainerStarted","Data":"46c05838ca215d7ebbbb02fccbc3ef661bdb71fb22500a22996f616dca1acb29"}
Nov 03 20:26:57.077983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.077853    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6" event={"ID":"0c16de55-ec16-4a5d-bb99-8793dd848e5b","Type":"ContainerStarted","Data":"fe202c225b90573f781a4fd23d982c7861f31b5cf97fbc5b6d55499f3207c8eb"}
Nov 03 20:26:57.094155 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.093643    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6" podStartSLOduration=2.093625536 podStartE2EDuration="2.093625536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:57.093347996 +0000 UTC m=+2799.606169778" watchObservedRunningTime="2026-11-03 20:26:57.093625536 +0000 UTC m=+2799.606447319"
Nov 03 20:26:57.235054 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.234957    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73c2e74-22gft\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:26:57.235054 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.235021    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73c2e74-22gft\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:26:57.235563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.235209    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbdpt\" (UniqueName: \"kubernetes.io/projected/69033edd-f107-4d62-874e-fb27cbe03b61-kube-api-access-kbdpt\") pod \"maas-api-revoke-sub-e73c2e74-22gft\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:26:57.336218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.336175    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73c2e74-22gft\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:26:57.336218 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.336220    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73c2e74-22gft\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:26:57.336472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.336260    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbdpt\" (UniqueName: \"kubernetes.io/projected/69033edd-f107-4d62-874e-fb27cbe03b61-kube-api-access-kbdpt\") pod \"maas-api-revoke-sub-e73c2e74-22gft\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:26:57.336919 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.336895    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73c2e74-22gft\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:26:57.338877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.338853    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73c2e74-22gft\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:26:57.344636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.344615    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbdpt\" (UniqueName: \"kubernetes.io/projected/69033edd-f107-4d62-874e-fb27cbe03b61-kube-api-access-kbdpt\") pod \"maas-api-revoke-sub-e73c2e74-22gft\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:26:57.374382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.374355    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:26:57.654929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:57.654899    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"]
Nov 03 20:26:57.658592 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:26:57.658557    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69033edd_f107_4d62_874e_fb27cbe03b61.slice/crio-887f4e17212ee8618e2e564649e2c3a59b9379d30e26854bfa18417ad9e9acdc WatchSource:0}: Error finding container 887f4e17212ee8618e2e564649e2c3a59b9379d30e26854bfa18417ad9e9acdc: Status 404 returned error can't find the container with id 887f4e17212ee8618e2e564649e2c3a59b9379d30e26854bfa18417ad9e9acdc
Nov 03 20:26:58.086778 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:58.086737    5600 generic.go:356] "Generic (PLEG): container finished" podID="0c16de55-ec16-4a5d-bb99-8793dd848e5b" containerID="46c05838ca215d7ebbbb02fccbc3ef661bdb71fb22500a22996f616dca1acb29" exitCode=0
Nov 03 20:26:58.086973 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:58.086877    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6" event={"ID":"0c16de55-ec16-4a5d-bb99-8793dd848e5b","Type":"ContainerDied","Data":"46c05838ca215d7ebbbb02fccbc3ef661bdb71fb22500a22996f616dca1acb29"}
Nov 03 20:26:58.088827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:58.088804    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft" event={"ID":"69033edd-f107-4d62-874e-fb27cbe03b61","Type":"ContainerStarted","Data":"9359290a817b482dca699351688522824e1d90e39427a03b76672f19bb4042a4"}
Nov 03 20:26:58.088944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:58.088852    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft" event={"ID":"69033edd-f107-4d62-874e-fb27cbe03b61","Type":"ContainerStarted","Data":"887f4e17212ee8618e2e564649e2c3a59b9379d30e26854bfa18417ad9e9acdc"}
Nov 03 20:26:58.118110 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:58.118047    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft" podStartSLOduration=1.118033637 podStartE2EDuration="1.118033637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:58.117768838 +0000 UTC m=+2800.630590622" watchObservedRunningTime="2026-11-03 20:26:58.118033637 +0000 UTC m=+2800.630855418"
Nov 03 20:26:58.800137 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:26:58.800093    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:26:59.099580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.099535    5600 generic.go:356] "Generic (PLEG): container finished" podID="69033edd-f107-4d62-874e-fb27cbe03b61" containerID="9359290a817b482dca699351688522824e1d90e39427a03b76672f19bb4042a4" exitCode=0
Nov 03 20:26:59.099888 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.099852    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft" event={"ID":"69033edd-f107-4d62-874e-fb27cbe03b61","Type":"ContainerDied","Data":"9359290a817b482dca699351688522824e1d90e39427a03b76672f19bb4042a4"}
Nov 03 20:26:59.253630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.253606    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:26:59.357646 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.357555    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-openshift-ca\") pod \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") "
Nov 03 20:26:59.357646 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.357636    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-xks-ca\") pod \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") "
Nov 03 20:26:59.357894 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.357752    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c16de55-ec16-4a5d-bb99-8793dd848e5b-kube-api-access-dpp2g\" (UniqueName: \"kubernetes.io/projected/0c16de55-ec16-4a5d-bb99-8793dd848e5b-kube-api-access-dpp2g\") pod \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\" (UID: \"0c16de55-ec16-4a5d-bb99-8793dd848e5b\") "
Nov 03 20:26:59.358037 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.358013    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-openshift-ca" pod "0c16de55-ec16-4a5d-bb99-8793dd848e5b" (UID: "0c16de55-ec16-4a5d-bb99-8793dd848e5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:59.358189 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.358173    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:59.359961 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.359941    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c16de55-ec16-4a5d-bb99-8793dd848e5b-kube-api-access-dpp2g" pod "0c16de55-ec16-4a5d-bb99-8793dd848e5b" (UID: "0c16de55-ec16-4a5d-bb99-8793dd848e5b"). InnerVolumeSpecName "kube-api-access-dpp2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:59.360058 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.360043    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-xks-ca" pod "0c16de55-ec16-4a5d-bb99-8793dd848e5b" (UID: "0c16de55-ec16-4a5d-bb99-8793dd848e5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:59.459640 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.459601    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpp2g\" (UniqueName: \"kubernetes.io/projected/0c16de55-ec16-4a5d-bb99-8793dd848e5b-kube-api-access-dpp2g\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:59.459640 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:26:59.459634    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c16de55-ec16-4a5d-bb99-8793dd848e5b-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.107890 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.107708    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj" podUID="4683c83d-c95b-4878-9802-81982c339032" containerName="payload-processing" containerID="cri-o://4b9f8de91739b7bb52f6a61c41ceb7d831c154669454ff0c831729e7a0de21d5" gracePeriod=25
Nov 03 20:27:00.119277 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.117372    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"
Nov 03 20:27:00.119277 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.118063    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6" event={"ID":"0c16de55-ec16-4a5d-bb99-8793dd848e5b","Type":"ContainerDied","Data":"fe202c225b90573f781a4fd23d982c7861f31b5cf97fbc5b6d55499f3207c8eb"}
Nov 03 20:27:00.119277 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.118110    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe202c225b90573f781a4fd23d982c7861f31b5cf97fbc5b6d55499f3207c8eb"
Nov 03 20:27:00.294687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.294657    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:27:00.470033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.469997    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-xks-ca\") pod \"69033edd-f107-4d62-874e-fb27cbe03b61\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") "
Nov 03 20:27:00.470269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.470064    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69033edd-f107-4d62-874e-fb27cbe03b61-kube-api-access-kbdpt\" (UniqueName: \"kubernetes.io/projected/69033edd-f107-4d62-874e-fb27cbe03b61-kube-api-access-kbdpt\") pod \"69033edd-f107-4d62-874e-fb27cbe03b61\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") "
Nov 03 20:27:00.470269 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.470120    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-openshift-ca\") pod \"69033edd-f107-4d62-874e-fb27cbe03b61\" (UID: \"69033edd-f107-4d62-874e-fb27cbe03b61\") "
Nov 03 20:27:00.470624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.470593    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-openshift-ca" pod "69033edd-f107-4d62-874e-fb27cbe03b61" (UID: "69033edd-f107-4d62-874e-fb27cbe03b61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:00.472492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.472425    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69033edd-f107-4d62-874e-fb27cbe03b61-kube-api-access-kbdpt" pod "69033edd-f107-4d62-874e-fb27cbe03b61" (UID: "69033edd-f107-4d62-874e-fb27cbe03b61"). InnerVolumeSpecName "kube-api-access-kbdpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:00.472597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.472512    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-xks-ca" pod "69033edd-f107-4d62-874e-fb27cbe03b61" (UID: "69033edd-f107-4d62-874e-fb27cbe03b61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:00.571323 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.571292    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.571323 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.571321    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbdpt\" (UniqueName: \"kubernetes.io/projected/69033edd-f107-4d62-874e-fb27cbe03b61-kube-api-access-kbdpt\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.571323 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:00.571331    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69033edd-f107-4d62-874e-fb27cbe03b61-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:01.128402 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:01.128357    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft" event={"ID":"69033edd-f107-4d62-874e-fb27cbe03b61","Type":"ContainerDied","Data":"887f4e17212ee8618e2e564649e2c3a59b9379d30e26854bfa18417ad9e9acdc"}
Nov 03 20:27:01.128961 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:01.128427    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="887f4e17212ee8618e2e564649e2c3a59b9379d30e26854bfa18417ad9e9acdc"
Nov 03 20:27:01.128961 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:01.128547    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"
Nov 03 20:27:02.726807 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:02.726760    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj" podUID="4683c83d-c95b-4878-9802-81982c339032" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:27:03.020385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.020258    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main" containerID="cri-o://3b63cb68ca93db40ff17d9f93882377f6d3627e9a1bdb9e4f498c511ccb5276e" gracePeriod=45
Nov 03 20:27:03.141674 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.141645    5600 generic.go:356] "Generic (PLEG): container finished" podID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerID="3b63cb68ca93db40ff17d9f93882377f6d3627e9a1bdb9e4f498c511ccb5276e" exitCode=0
Nov 03 20:27:03.141818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.141719    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" event={"ID":"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34","Type":"ContainerDied","Data":"3b63cb68ca93db40ff17d9f93882377f6d3627e9a1bdb9e4f498c511ccb5276e"}
Nov 03 20:27:03.295856 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.295827    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:27:03.302843 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.302811    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-home\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-home\") pod \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") "
Nov 03 20:27:03.303003 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.302906    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tmp-dir\") pod \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") "
Nov 03 20:27:03.303083 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.303007    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-dshm\") pod \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") "
Nov 03 20:27:03.303169 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.303112    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-home" pod "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" (UID: "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.303227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.303192    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tmp-dir" pod "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" (UID: "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.303408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.303391    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.303483 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.303415    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.305482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.305456    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-dshm" pod "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" (UID: "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.404031 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.403985    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-kube-api-access-k6f8z\" (UniqueName: \"kubernetes.io/projected/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-kube-api-access-k6f8z\") pod \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") "
Nov 03 20:27:03.404178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.404053    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tls-certs\") pod \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") "
Nov 03 20:27:03.404178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.404091    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-model-cache\") pod \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\" (UID: \"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34\") "
Nov 03 20:27:03.404448 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.404419    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.405000 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.404976    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-model-cache" pod "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" (UID: "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.418656 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.413684    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-kube-api-access-k6f8z" pod "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" (UID: "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34"). InnerVolumeSpecName "kube-api-access-k6f8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:03.418656 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.413995    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tls-certs" pod "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" (UID: "fe85ef7f-2ad1-4707-8a1e-00075b3e3f34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:03.505523 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.505483    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6f8z\" (UniqueName: \"kubernetes.io/projected/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-kube-api-access-k6f8z\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.505523 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.505523    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.505759 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:03.505539    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:04.149127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:04.149089    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs" event={"ID":"fe85ef7f-2ad1-4707-8a1e-00075b3e3f34","Type":"ContainerDied","Data":"b72d234c9eb2f436aefcddb45cf885babfb52a1b112f0e06cb3998a37a0ab1b9"}
Nov 03 20:27:04.149663 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:04.149136    5600 scope.go:122] "RemoveContainer" containerID="3b63cb68ca93db40ff17d9f93882377f6d3627e9a1bdb9e4f498c511ccb5276e"
Nov 03 20:27:04.149663 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:04.149095    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"
Nov 03 20:27:04.166058 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:04.166029    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"]
Nov 03 20:27:04.171653 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:04.171632    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-2-w5-ab1e46-kserve-54744c8876-wk7cs"]
Nov 03 20:27:05.616246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:05.616209    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" path="/var/lib/kubelet/pods/fe85ef7f-2ad1-4707-8a1e-00075b3e3f34/volumes"
Nov 03 20:27:06.491808 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:06.491773    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"]
Nov 03 20:27:06.492553 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:06.492507    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp" podUID="d8868e28-6951-4711-88ba-498cf360042e" containerName="maas-api" containerID="cri-o://272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b" gracePeriod=30
Nov 03 20:27:06.653177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:06.653133    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp" podUID="d8868e28-6951-4711-88ba-498cf360042e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.155:8443/health\": dial tcp 10.134.0.155:8443: connect: connection refused"
Nov 03 20:27:06.763771 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:06.763677    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"]
Nov 03 20:27:06.764316 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:06.764293    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b55105-ln2w6"]
Nov 03 20:27:07.538528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.538490    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"]
Nov 03 20:27:07.539145 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.539109    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn" podUID="63df7cb9-548d-40c6-ab66-db4bb639501f" containerName="maas-api" containerID="cri-o://ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781" gracePeriod=30
Nov 03 20:27:07.632468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.632390    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c16de55-ec16-4a5d-bb99-8793dd848e5b" path="/var/lib/kubelet/pods/0c16de55-ec16-4a5d-bb99-8793dd848e5b/volumes"
Nov 03 20:27:07.725717 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.725417    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn" podUID="63df7cb9-548d-40c6-ab66-db4bb639501f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused"
Nov 03 20:27:07.932687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.932657    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:27:07.960514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.960482    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8868e28-6951-4711-88ba-498cf360042e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8868e28-6951-4711-88ba-498cf360042e-openshift-service-ca\") pod \"d8868e28-6951-4711-88ba-498cf360042e\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") "
Nov 03 20:27:07.960687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.960582    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-metrics-certs\") pod \"d8868e28-6951-4711-88ba-498cf360042e\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") "
Nov 03 20:27:07.960687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.960627    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-maas-api-tls\") pod \"d8868e28-6951-4711-88ba-498cf360042e\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") "
Nov 03 20:27:07.960824 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.960690    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8868e28-6951-4711-88ba-498cf360042e-kube-api-access-k622n\" (UniqueName: \"kubernetes.io/projected/d8868e28-6951-4711-88ba-498cf360042e-kube-api-access-k622n\") pod \"d8868e28-6951-4711-88ba-498cf360042e\" (UID: \"d8868e28-6951-4711-88ba-498cf360042e\") "
Nov 03 20:27:07.960886 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.960860    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8868e28-6951-4711-88ba-498cf360042e-openshift-service-ca" pod "d8868e28-6951-4711-88ba-498cf360042e" (UID: "d8868e28-6951-4711-88ba-498cf360042e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:07.961020 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.960999    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8868e28-6951-4711-88ba-498cf360042e-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:07.962995 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.962970    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8868e28-6951-4711-88ba-498cf360042e-kube-api-access-k622n" pod "d8868e28-6951-4711-88ba-498cf360042e" (UID: "d8868e28-6951-4711-88ba-498cf360042e"). InnerVolumeSpecName "kube-api-access-k622n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:07.963122 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.963002    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-maas-api-tls" pod "d8868e28-6951-4711-88ba-498cf360042e" (UID: "d8868e28-6951-4711-88ba-498cf360042e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:07.963122 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.963069    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-metrics-certs" pod "d8868e28-6951-4711-88ba-498cf360042e" (UID: "d8868e28-6951-4711-88ba-498cf360042e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:07.971421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:07.971385    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:27:08.062017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.061943    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63df7cb9-548d-40c6-ab66-db4bb639501f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63df7cb9-548d-40c6-ab66-db4bb639501f-openshift-service-ca\") pod \"63df7cb9-548d-40c6-ab66-db4bb639501f\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") "
Nov 03 20:27:08.062017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.061995    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-maas-api-tls\") pod \"63df7cb9-548d-40c6-ab66-db4bb639501f\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") "
Nov 03 20:27:08.062243 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.062032    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-metrics-certs\") pod \"63df7cb9-548d-40c6-ab66-db4bb639501f\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") "
Nov 03 20:27:08.062243 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.062216    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63df7cb9-548d-40c6-ab66-db4bb639501f-kube-api-access-c625f\" (UniqueName: \"kubernetes.io/projected/63df7cb9-548d-40c6-ab66-db4bb639501f-kube-api-access-c625f\") pod \"63df7cb9-548d-40c6-ab66-db4bb639501f\" (UID: \"63df7cb9-548d-40c6-ab66-db4bb639501f\") "
Nov 03 20:27:08.062464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.062404    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63df7cb9-548d-40c6-ab66-db4bb639501f-openshift-service-ca" pod "63df7cb9-548d-40c6-ab66-db4bb639501f" (UID: "63df7cb9-548d-40c6-ab66-db4bb639501f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:08.062682 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.062663    5600 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63df7cb9-548d-40c6-ab66-db4bb639501f-openshift-service-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:08.062747 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.062689    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:08.062747 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.062702    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8868e28-6951-4711-88ba-498cf360042e-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:08.062747 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.062712    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k622n\" (UniqueName: \"kubernetes.io/projected/d8868e28-6951-4711-88ba-498cf360042e-kube-api-access-k622n\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:08.064283 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.064258    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-metrics-certs" pod "63df7cb9-548d-40c6-ab66-db4bb639501f" (UID: "63df7cb9-548d-40c6-ab66-db4bb639501f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:08.064283 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.064275    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-maas-api-tls" pod "63df7cb9-548d-40c6-ab66-db4bb639501f" (UID: "63df7cb9-548d-40c6-ab66-db4bb639501f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:08.064404 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.064381    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63df7cb9-548d-40c6-ab66-db4bb639501f-kube-api-access-c625f" pod "63df7cb9-548d-40c6-ab66-db4bb639501f" (UID: "63df7cb9-548d-40c6-ab66-db4bb639501f"). InnerVolumeSpecName "kube-api-access-c625f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.167847    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"]
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.168227    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy" containerID="cri-o://3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8" gracePeriod=30
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.171191    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-maas-api-tls\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.171215    5600 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63df7cb9-548d-40c6-ab66-db4bb639501f-metrics-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.171231    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c625f\" (UniqueName: \"kubernetes.io/projected/63df7cb9-548d-40c6-ab66-db4bb639501f-kube-api-access-c625f\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.182757    5600 generic.go:356] "Generic (PLEG): container finished" podID="d8868e28-6951-4711-88ba-498cf360042e" containerID="272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b" exitCode=0
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.182864    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp" event={"ID":"d8868e28-6951-4711-88ba-498cf360042e","Type":"ContainerDied","Data":"272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b"}
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.182912    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp" event={"ID":"d8868e28-6951-4711-88ba-498cf360042e","Type":"ContainerDied","Data":"ee548cf036c59d1dfe1223dc27be63e2b799f09120c7d92b7ee5bc80fd24233e"}
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.182942    5600 scope.go:122] "RemoveContainer" containerID="272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b"
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.183083    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.189232    5600 generic.go:356] "Generic (PLEG): container finished" podID="63df7cb9-548d-40c6-ab66-db4bb639501f" containerID="ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781" exitCode=0
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.189367    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn" event={"ID":"63df7cb9-548d-40c6-ab66-db4bb639501f","Type":"ContainerDied","Data":"ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781"}
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.189402    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn" event={"ID":"63df7cb9-548d-40c6-ab66-db4bb639501f","Type":"ContainerDied","Data":"aad78c6e2f19042cca87718fd5d4c908afd8d9b48b89511afb04456a99d7fcc6"}
Nov 03 20:27:08.197306 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.189602    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"
Nov 03 20:27:08.217256 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.217224    5600 scope.go:122] "RemoveContainer" containerID="272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b"
Nov 03 20:27:08.217967 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:27:08.217923    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b\": container with ID starting with 272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b not found: ID does not exist" containerID="272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b"
Nov 03 20:27:08.218090 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.218001    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b"} err="failed to get container status \"272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b\": rpc error: code = NotFound desc = could not find container \"272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b\": container with ID starting with 272333e23ec85d03ef99368dce3b2d8fda0bc335fa0853956bc50327c4356b4b not found: ID does not exist"
Nov 03 20:27:08.218090 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.218028    5600 scope.go:122] "RemoveContainer" containerID="ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781"
Nov 03 20:27:08.221195 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.221169    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"]
Nov 03 20:27:08.226750 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.226727    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b55105-555d69c4f9-7vsmp"]
Nov 03 20:27:08.238387 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.238358    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"]
Nov 03 20:27:08.241609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.241583    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ab1e46-5b6b7d6979-249jn"]
Nov 03 20:27:08.243136 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.243112    5600 scope.go:122] "RemoveContainer" containerID="ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781"
Nov 03 20:27:08.243562 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:27:08.243529    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781\": container with ID starting with ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781 not found: ID does not exist" containerID="ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781"
Nov 03 20:27:08.243710 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:08.243671    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781"} err="failed to get container status \"ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781\": rpc error: code = NotFound desc = could not find container \"ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781\": container with ID starting with ff3f60626e95cc9fb45d6ef24679fdaf87ecba7278a4760564989e1ef87d3781 not found: ID does not exist"
Nov 03 20:27:09.617687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:09.617642    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63df7cb9-548d-40c6-ab66-db4bb639501f" path="/var/lib/kubelet/pods/63df7cb9-548d-40c6-ab66-db4bb639501f/volumes"
Nov 03 20:27:09.618259 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:09.618240    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8868e28-6951-4711-88ba-498cf360042e" path="/var/lib/kubelet/pods/d8868e28-6951-4711-88ba-498cf360042e/volumes"
Nov 03 20:27:12.726826 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:12.726788    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj" podUID="4683c83d-c95b-4878-9802-81982c339032" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:27:13.480201 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.480174    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:27:13.548492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.533877    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-data\") pod \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") "
Nov 03 20:27:13.548492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.533947    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-envoy\") pod \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") "
Nov 03 20:27:13.548492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.533991    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-kube-api-access-r69p7\" (UniqueName: \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-kube-api-access-r69p7\") pod \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") "
Nov 03 20:27:13.548492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.534046    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-certs\") pod \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") "
Nov 03 20:27:13.548492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.534089    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-podinfo\") pod \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") "
Nov 03 20:27:13.548492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.534156    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istiod-ca-cert\") pod \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") "
Nov 03 20:27:13.548492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.534192    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-socket\") pod \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") "
Nov 03 20:27:13.548492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.534262    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-credential-socket\") pod \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") "
Nov 03 20:27:13.548492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.534333    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-token\" (UniqueName: \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-token\") pod \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\" (UID: \"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d\") "
Nov 03 20:27:13.548492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.547909    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-certs" pod "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" (UID: "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:13.548492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.548118    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-data" pod "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" (UID: "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:13.551947 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.551618    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-socket" pod "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" (UID: "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:13.552878 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.552843    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istiod-ca-cert" pod "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" (UID: "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:13.553021 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.552914    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-envoy" pod "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" (UID: "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:13.553289 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.553253    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-credential-socket" pod "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" (UID: "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:13.559538 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.557465    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-token" pod "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" (UID: "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:13.559538 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.559033    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-kube-api-access-r69p7" pod "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" (UID: "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d"). InnerVolumeSpecName "kube-api-access-r69p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:13.559538 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.559197    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-podinfo" pod "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" (UID: "ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:13.635953 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.635924    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.635953 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.635950    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.636127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.635962    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r69p7\" (UniqueName: \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-kube-api-access-r69p7\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.636127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.635981    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.636127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.635991    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.636127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.636000    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.636127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.636009    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.636127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.636018    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.636127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:13.636025    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:14.237974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:14.237932    5600 generic.go:356] "Generic (PLEG): container finished" podID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerID="3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8" exitCode=0
Nov 03 20:27:14.237974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:14.237986    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" event={"ID":"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d","Type":"ContainerDied","Data":"3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8"}
Nov 03 20:27:14.237974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:14.238000    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"
Nov 03 20:27:14.238677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:14.238042    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248" event={"ID":"ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d","Type":"ContainerDied","Data":"2e6274eb982790203ebb0abcf83b892171494412fd0195e2a15a002e7fa1327e"}
Nov 03 20:27:14.238677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:14.238091    5600 scope.go:122] "RemoveContainer" containerID="3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8"
Nov 03 20:27:14.248676 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:14.248653    5600 scope.go:122] "RemoveContainer" containerID="3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8"
Nov 03 20:27:14.248994 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:27:14.248974    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8\": container with ID starting with 3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8 not found: ID does not exist" containerID="3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8"
Nov 03 20:27:14.249046 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:14.249002    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8"} err="failed to get container status \"3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8\": rpc error: code = NotFound desc = could not find container \"3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8\": container with ID starting with 3555b4f24260381faae515582b2e543a0b15e952d466fe14ec2b339304020fc8 not found: ID does not exist"
Nov 03 20:27:14.257857 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:14.257827    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"]
Nov 03 20:27:14.261059 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:14.261029    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-dj248"]
Nov 03 20:27:15.168671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:15.168636    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"]
Nov 03 20:27:15.168900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:15.168874    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h" podUID="da904e7b-25ba-4397-9841-61cccacfc4ae" containerName="istio-proxy" containerID="cri-o://715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972" gracePeriod=30
Nov 03 20:27:15.616173 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:15.616090    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" path="/var/lib/kubelet/pods/ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d/volumes"
Nov 03 20:27:17.384645 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:17.384576    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"]
Nov 03 20:27:17.387242 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:17.385131    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy" containerID="cri-o://670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f" gracePeriod=30
Nov 03 20:27:18.989482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:18.989253    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"]
Nov 03 20:27:18.990021 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:18.989600    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerName="istio-proxy" containerID="cri-o://43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222" gracePeriod=30
Nov 03 20:27:18.992681 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:18.992656    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"]
Nov 03 20:27:18.993096 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:18.993065    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy" containerID="cri-o://dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a" gracePeriod=30
Nov 03 20:27:19.524043 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:19.524005    5600 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:19.524235 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:19.524066    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:20.279412 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.279386    5600 generic.go:356] "Generic (PLEG): container finished" podID="4683c83d-c95b-4878-9802-81982c339032" containerID="4b9f8de91739b7bb52f6a61c41ceb7d831c154669454ff0c831729e7a0de21d5" exitCode=0
Nov 03 20:27:20.279717 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.279461    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj" event={"ID":"4683c83d-c95b-4878-9802-81982c339032","Type":"ContainerDied","Data":"4b9f8de91739b7bb52f6a61c41ceb7d831c154669454ff0c831729e7a0de21d5"}
Nov 03 20:27:20.457202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.457168    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:27:20.493529 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.493499    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:27:20.519420 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.519348    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4683c83d-c95b-4878-9802-81982c339032-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4683c83d-c95b-4878-9802-81982c339032-plugins-config-volume\") pod \"4683c83d-c95b-4878-9802-81982c339032\" (UID: \"4683c83d-c95b-4878-9802-81982c339032\") "
Nov 03 20:27:20.520165 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.519736    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4683c83d-c95b-4878-9802-81982c339032-kube-api-access-z9nzg\" (UniqueName: \"kubernetes.io/projected/4683c83d-c95b-4878-9802-81982c339032-kube-api-access-z9nzg\") pod \"4683c83d-c95b-4878-9802-81982c339032\" (UID: \"4683c83d-c95b-4878-9802-81982c339032\") "
Nov 03 20:27:20.520282 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.520226    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4683c83d-c95b-4878-9802-81982c339032-plugins-config-volume" pod "4683c83d-c95b-4878-9802-81982c339032" (UID: "4683c83d-c95b-4878-9802-81982c339032"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:20.524031 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.524000    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4683c83d-c95b-4878-9802-81982c339032-kube-api-access-z9nzg" pod "4683c83d-c95b-4878-9802-81982c339032" (UID: "4683c83d-c95b-4878-9802-81982c339032"). InnerVolumeSpecName "kube-api-access-z9nzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:20.620908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.620874    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-certs\") pod \"da904e7b-25ba-4397-9841-61cccacfc4ae\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") "
Nov 03 20:27:20.621106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.620916    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-token\" (UniqueName: \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-token\") pod \"da904e7b-25ba-4397-9841-61cccacfc4ae\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") "
Nov 03 20:27:20.621106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.620966    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-kube-api-access-wzhv8\" (UniqueName: \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-kube-api-access-wzhv8\") pod \"da904e7b-25ba-4397-9841-61cccacfc4ae\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") "
Nov 03 20:27:20.621106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621040    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-credential-socket\") pod \"da904e7b-25ba-4397-9841-61cccacfc4ae\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") "
Nov 03 20:27:20.621106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621067    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-socket\") pod \"da904e7b-25ba-4397-9841-61cccacfc4ae\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") "
Nov 03 20:27:20.621833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621131    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-podinfo\") pod \"da904e7b-25ba-4397-9841-61cccacfc4ae\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") "
Nov 03 20:27:20.621833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621190    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da904e7b-25ba-4397-9841-61cccacfc4ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da904e7b-25ba-4397-9841-61cccacfc4ae-istiod-ca-cert\") pod \"da904e7b-25ba-4397-9841-61cccacfc4ae\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") "
Nov 03 20:27:20.621833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621231    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-envoy\") pod \"da904e7b-25ba-4397-9841-61cccacfc4ae\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") "
Nov 03 20:27:20.621833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621267    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-data\") pod \"da904e7b-25ba-4397-9841-61cccacfc4ae\" (UID: \"da904e7b-25ba-4397-9841-61cccacfc4ae\") "
Nov 03 20:27:20.621833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621276    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-certs" pod "da904e7b-25ba-4397-9841-61cccacfc4ae" (UID: "da904e7b-25ba-4397-9841-61cccacfc4ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.621833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621682    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.621833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621707    5600 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4683c83d-c95b-4878-9802-81982c339032-plugins-config-volume\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.621833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621709    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-socket" pod "da904e7b-25ba-4397-9841-61cccacfc4ae" (UID: "da904e7b-25ba-4397-9841-61cccacfc4ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.621833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621723    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9nzg\" (UniqueName: \"kubernetes.io/projected/4683c83d-c95b-4878-9802-81982c339032-kube-api-access-z9nzg\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.622525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621911    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-data" pod "da904e7b-25ba-4397-9841-61cccacfc4ae" (UID: "da904e7b-25ba-4397-9841-61cccacfc4ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.622525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.621930    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-credential-socket" pod "da904e7b-25ba-4397-9841-61cccacfc4ae" (UID: "da904e7b-25ba-4397-9841-61cccacfc4ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.622525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.622156    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da904e7b-25ba-4397-9841-61cccacfc4ae-istiod-ca-cert" pod "da904e7b-25ba-4397-9841-61cccacfc4ae" (UID: "da904e7b-25ba-4397-9841-61cccacfc4ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:20.623847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.623821    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-token" pod "da904e7b-25ba-4397-9841-61cccacfc4ae" (UID: "da904e7b-25ba-4397-9841-61cccacfc4ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:20.624003 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.623978    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-podinfo" pod "da904e7b-25ba-4397-9841-61cccacfc4ae" (UID: "da904e7b-25ba-4397-9841-61cccacfc4ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:20.624085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.624053    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-kube-api-access-wzhv8" pod "da904e7b-25ba-4397-9841-61cccacfc4ae" (UID: "da904e7b-25ba-4397-9841-61cccacfc4ae"). InnerVolumeSpecName "kube-api-access-wzhv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:20.624228 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.624213    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-envoy" pod "da904e7b-25ba-4397-9841-61cccacfc4ae" (UID: "da904e7b-25ba-4397-9841-61cccacfc4ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.722679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.722600    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.722679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.722632    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzhv8\" (UniqueName: \"kubernetes.io/projected/da904e7b-25ba-4397-9841-61cccacfc4ae-kube-api-access-wzhv8\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.722679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.722642    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.722679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.722651    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.722679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.722661    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.722679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.722669    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da904e7b-25ba-4397-9841-61cccacfc4ae-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.722679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.722678    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.722679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:20.722685    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da904e7b-25ba-4397-9841-61cccacfc4ae-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:21.286866 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.286826    5600 generic.go:356] "Generic (PLEG): container finished" podID="da904e7b-25ba-4397-9841-61cccacfc4ae" containerID="715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972" exitCode=0
Nov 03 20:27:21.287400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.286881    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"
Nov 03 20:27:21.287400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.286906    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h" event={"ID":"da904e7b-25ba-4397-9841-61cccacfc4ae","Type":"ContainerDied","Data":"715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972"}
Nov 03 20:27:21.287400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.286982    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h" event={"ID":"da904e7b-25ba-4397-9841-61cccacfc4ae","Type":"ContainerDied","Data":"4c04493906cb2d31529eeacd8856f7b6f50bbb65ab0e3d25c87dd8e131d5a3d5"}
Nov 03 20:27:21.287400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.287013    5600 scope.go:122] "RemoveContainer" containerID="715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972"
Nov 03 20:27:21.289122 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.289094    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj" event={"ID":"4683c83d-c95b-4878-9802-81982c339032","Type":"ContainerDied","Data":"70e0c511fea82d481e871d31fe95a058da1faaaeac3076760c20e5f27c344cbc"}
Nov 03 20:27:21.289122 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.289123    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"
Nov 03 20:27:21.299367 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.299349    5600 scope.go:122] "RemoveContainer" containerID="715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972"
Nov 03 20:27:21.299843 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:27:21.299814    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972\": container with ID starting with 715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972 not found: ID does not exist" containerID="715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972"
Nov 03 20:27:21.299944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.299865    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972"} err="failed to get container status \"715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972\": rpc error: code = NotFound desc = could not find container \"715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972\": container with ID starting with 715ae05b19b247d8249eb1e3209888c4b530cb6729f139a7ee9ed696c8131972 not found: ID does not exist"
Nov 03 20:27:21.299944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.299890    5600 scope.go:122] "RemoveContainer" containerID="4b9f8de91739b7bb52f6a61c41ceb7d831c154669454ff0c831729e7a0de21d5"
Nov 03 20:27:21.316064 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.316033    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"]
Nov 03 20:27:21.319837 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.319812    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-g426h"]
Nov 03 20:27:21.331697 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.331667    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"]
Nov 03 20:27:21.335355 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.335332    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b42aea-5b857c86-d7swj"]
Nov 03 20:27:21.616013 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.615980    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4683c83d-c95b-4878-9802-81982c339032" path="/var/lib/kubelet/pods/4683c83d-c95b-4878-9802-81982c339032/volumes"
Nov 03 20:27:21.616482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:21.616428    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da904e7b-25ba-4397-9841-61cccacfc4ae" path="/var/lib/kubelet/pods/da904e7b-25ba-4397-9841-61cccacfc4ae/volumes"
Nov 03 20:27:22.656071 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.656047    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:27:22.744086 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.743998    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-podinfo\") pod \"4e0e925b-4187-4596-8098-6a660bdb13a8\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") "
Nov 03 20:27:22.744234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744090    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-socket\") pod \"4e0e925b-4187-4596-8098-6a660bdb13a8\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") "
Nov 03 20:27:22.744234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744141    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-token\" (UniqueName: \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-token\") pod \"4e0e925b-4187-4596-8098-6a660bdb13a8\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") "
Nov 03 20:27:22.744234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744164    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-envoy\") pod \"4e0e925b-4187-4596-8098-6a660bdb13a8\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") "
Nov 03 20:27:22.744234 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744206    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-data\") pod \"4e0e925b-4187-4596-8098-6a660bdb13a8\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") "
Nov 03 20:27:22.744508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744244    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-credential-socket\") pod \"4e0e925b-4187-4596-8098-6a660bdb13a8\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") "
Nov 03 20:27:22.744508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744285    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-certs\") pod \"4e0e925b-4187-4596-8098-6a660bdb13a8\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") "
Nov 03 20:27:22.744508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744328    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-kube-api-access-pmlhd\" (UniqueName: \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-kube-api-access-pmlhd\") pod \"4e0e925b-4187-4596-8098-6a660bdb13a8\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") "
Nov 03 20:27:22.744508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744369    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e0e925b-4187-4596-8098-6a660bdb13a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0e925b-4187-4596-8098-6a660bdb13a8-istiod-ca-cert\") pod \"4e0e925b-4187-4596-8098-6a660bdb13a8\" (UID: \"4e0e925b-4187-4596-8098-6a660bdb13a8\") "
Nov 03 20:27:22.744508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744477    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-socket" pod "4e0e925b-4187-4596-8098-6a660bdb13a8" (UID: "4e0e925b-4187-4596-8098-6a660bdb13a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:22.744772 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744592    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-data" pod "4e0e925b-4187-4596-8098-6a660bdb13a8" (UID: "4e0e925b-4187-4596-8098-6a660bdb13a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:22.744838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744809    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-certs" pod "4e0e925b-4187-4596-8098-6a660bdb13a8" (UID: "4e0e925b-4187-4596-8098-6a660bdb13a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:22.744838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744826    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.744934 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744848    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.744934 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744826    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0e925b-4187-4596-8098-6a660bdb13a8-istiod-ca-cert" pod "4e0e925b-4187-4596-8098-6a660bdb13a8" (UID: "4e0e925b-4187-4596-8098-6a660bdb13a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:22.745022 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.744989    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-credential-socket" pod "4e0e925b-4187-4596-8098-6a660bdb13a8" (UID: "4e0e925b-4187-4596-8098-6a660bdb13a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:22.746681 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.746651    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-token" pod "4e0e925b-4187-4596-8098-6a660bdb13a8" (UID: "4e0e925b-4187-4596-8098-6a660bdb13a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:22.747107 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.747077    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-podinfo" pod "4e0e925b-4187-4596-8098-6a660bdb13a8" (UID: "4e0e925b-4187-4596-8098-6a660bdb13a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:22.747210 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.747192    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-envoy" pod "4e0e925b-4187-4596-8098-6a660bdb13a8" (UID: "4e0e925b-4187-4596-8098-6a660bdb13a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:22.747210 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.747194    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-kube-api-access-pmlhd" pod "4e0e925b-4187-4596-8098-6a660bdb13a8" (UID: "4e0e925b-4187-4596-8098-6a660bdb13a8"). InnerVolumeSpecName "kube-api-access-pmlhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:22.845643 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.845610    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmlhd\" (UniqueName: \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-kube-api-access-pmlhd\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.845643 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.845638    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0e925b-4187-4596-8098-6a660bdb13a8-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.845643 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.845647    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.845958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.845658    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.845958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.845668    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.845958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.845676    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.845958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:22.845684    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0e925b-4187-4596-8098-6a660bdb13a8-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:23.301595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.301556    5600 generic.go:356] "Generic (PLEG): container finished" podID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerID="670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f" exitCode=0
Nov 03 20:27:23.301788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.301602    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"
Nov 03 20:27:23.301788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.301643    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" event={"ID":"4e0e925b-4187-4596-8098-6a660bdb13a8","Type":"ContainerDied","Data":"670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f"}
Nov 03 20:27:23.301788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.301711    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k" event={"ID":"4e0e925b-4187-4596-8098-6a660bdb13a8","Type":"ContainerDied","Data":"f819442cdb78bbd3c84298187c04ae83a5eb4cb6ff43bc9e0bc7fb798ec4c3ab"}
Nov 03 20:27:23.301788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.301741    5600 scope.go:122] "RemoveContainer" containerID="670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f"
Nov 03 20:27:23.313858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.313833    5600 scope.go:122] "RemoveContainer" containerID="29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0"
Nov 03 20:27:23.325175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.325150    5600 scope.go:122] "RemoveContainer" containerID="670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f"
Nov 03 20:27:23.325579 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:27:23.325553    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f\": container with ID starting with 670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f not found: ID does not exist" containerID="670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f"
Nov 03 20:27:23.325691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.325592    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f"} err="failed to get container status \"670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f\": rpc error: code = NotFound desc = could not find container \"670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f\": container with ID starting with 670700fbcb9aa9e3bcd3a96777afa1e9853ddc223c12d676a2e07f7c87a5775f not found: ID does not exist"
Nov 03 20:27:23.325691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.325617    5600 scope.go:122] "RemoveContainer" containerID="29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0"
Nov 03 20:27:23.325927 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:27:23.325907    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0\": container with ID starting with 29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0 not found: ID does not exist" containerID="29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0"
Nov 03 20:27:23.325998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.325933    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0"} err="failed to get container status \"29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0\": rpc error: code = NotFound desc = could not find container \"29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0\": container with ID starting with 29c1188096db3285ced454e93037f5e9f3c313ffb49faa76a8ffeb3362c6dce0 not found: ID does not exist"
Nov 03 20:27:23.327675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.327648    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"]
Nov 03 20:27:23.332775 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.332754    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-hzf7k"]
Nov 03 20:27:23.615706 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:23.615629    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" path="/var/lib/kubelet/pods/4e0e925b-4187-4596-8098-6a660bdb13a8/volumes"
Nov 03 20:27:24.289194 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.289167    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:27:24.293354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.293333    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:27:24.308654 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.308622    5600 generic.go:356] "Generic (PLEG): container finished" podID="dcf21814-f278-425e-998e-a2448cb1fe12" containerID="dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a" exitCode=0
Nov 03 20:27:24.308817 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.308685    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"
Nov 03 20:27:24.308817 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.308721    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" event={"ID":"dcf21814-f278-425e-998e-a2448cb1fe12","Type":"ContainerDied","Data":"dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a"}
Nov 03 20:27:24.308817 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.308788    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7" event={"ID":"dcf21814-f278-425e-998e-a2448cb1fe12","Type":"ContainerDied","Data":"93be39bcd8ab643e2897ccffe7a3b4e65b87ff5f26fcafc8e56d65e0a145424c"}
Nov 03 20:27:24.308990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.308822    5600 scope.go:122] "RemoveContainer" containerID="dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a"
Nov 03 20:27:24.311845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.310393    5600 generic.go:356] "Generic (PLEG): container finished" podID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerID="43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222" exitCode=0
Nov 03 20:27:24.311845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.310546    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" event={"ID":"aca8cce9-d587-4466-b46d-3fa8bc0a8583","Type":"ContainerDied","Data":"43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222"}
Nov 03 20:27:24.311845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.310620    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m" event={"ID":"aca8cce9-d587-4466-b46d-3fa8bc0a8583","Type":"ContainerDied","Data":"fd5fe91920699f91120cb78fed2b979d8f6a32c4d5130aa0c13f2776dc70a89b"}
Nov 03 20:27:24.311845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.310717    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"
Nov 03 20:27:24.325200 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.325178    5600 scope.go:122] "RemoveContainer" containerID="dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a"
Nov 03 20:27:24.325637 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:27:24.325604    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a\": container with ID starting with dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a not found: ID does not exist" containerID="dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a"
Nov 03 20:27:24.325816 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.325647    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a"} err="failed to get container status \"dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a\": rpc error: code = NotFound desc = could not find container \"dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a\": container with ID starting with dc630ec3923f1828686dafaabc4e69332994edb144bd191cedbcfb8a1c3eb99a not found: ID does not exist"
Nov 03 20:27:24.325816 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.325671    5600 scope.go:122] "RemoveContainer" containerID="43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222"
Nov 03 20:27:24.357815 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.357688    5600 scope.go:122] "RemoveContainer" containerID="43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222"
Nov 03 20:27:24.358328 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:27:24.358121    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222\": container with ID starting with 43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222 not found: ID does not exist" containerID="43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222"
Nov 03 20:27:24.358328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.358170    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222"} err="failed to get container status \"43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222\": rpc error: code = NotFound desc = could not find container \"43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222\": container with ID starting with 43af9db4bb45b7c408704861309b9e334ed70d95ab8028b5268d451c47bfb222 not found: ID does not exist"
Nov 03 20:27:24.361678 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.361648    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf21814-f278-425e-998e-a2448cb1fe12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf21814-f278-425e-998e-a2448cb1fe12-istiod-ca-cert\") pod \"dcf21814-f278-425e-998e-a2448cb1fe12\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") "
Nov 03 20:27:24.361806 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.361706    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-certs\") pod \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") "
Nov 03 20:27:24.361806 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.361747    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-podinfo\") pod \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") "
Nov 03 20:27:24.361806 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.361782    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcf21814-f278-425e-998e-a2448cb1fe12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf21814-f278-425e-998e-a2448cb1fe12-istio-podinfo\") pod \"dcf21814-f278-425e-998e-a2448cb1fe12\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") "
Nov 03 20:27:24.361983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.361822    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-data\") pod \"dcf21814-f278-425e-998e-a2448cb1fe12\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") "
Nov 03 20:27:24.361983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.361859    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-envoy\") pod \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") "
Nov 03 20:27:24.361983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.361889    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istiod-ca-cert\") pod \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") "
Nov 03 20:27:24.361983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.361919    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-certs\") pod \"dcf21814-f278-425e-998e-a2448cb1fe12\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") "
Nov 03 20:27:24.362213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362004    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-token\" (UniqueName: \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-token\") pod \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") "
Nov 03 20:27:24.362213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362035    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-data\") pod \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") "
Nov 03 20:27:24.362213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362064    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf21814-f278-425e-998e-a2448cb1fe12-istiod-ca-cert" pod "dcf21814-f278-425e-998e-a2448cb1fe12" (UID: "dcf21814-f278-425e-998e-a2448cb1fe12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:24.362213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362116    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-kube-api-access-2r74f\" (UniqueName: \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-kube-api-access-2r74f\") pod \"dcf21814-f278-425e-998e-a2448cb1fe12\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") "
Nov 03 20:27:24.362213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362170    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-istio-token\" (UniqueName: \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-istio-token\") pod \"dcf21814-f278-425e-998e-a2448cb1fe12\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") "
Nov 03 20:27:24.362213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362205    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-kube-api-access-kp4t2\" (UniqueName: \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-kube-api-access-kp4t2\") pod \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") "
Nov 03 20:27:24.362627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362235    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-credential-socket\") pod \"dcf21814-f278-425e-998e-a2448cb1fe12\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") "
Nov 03 20:27:24.362627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362279    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-socket\") pod \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") "
Nov 03 20:27:24.362627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362320    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-credential-socket\") pod \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\" (UID: \"aca8cce9-d587-4466-b46d-3fa8bc0a8583\") "
Nov 03 20:27:24.362627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362368    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-socket\") pod \"dcf21814-f278-425e-998e-a2448cb1fe12\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") "
Nov 03 20:27:24.362627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362378    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-data" pod "dcf21814-f278-425e-998e-a2448cb1fe12" (UID: "dcf21814-f278-425e-998e-a2448cb1fe12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.362627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362397    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-envoy\") pod \"dcf21814-f278-425e-998e-a2448cb1fe12\" (UID: \"dcf21814-f278-425e-998e-a2448cb1fe12\") "
Nov 03 20:27:24.362627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362601    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-certs" pod "aca8cce9-d587-4466-b46d-3fa8bc0a8583" (UID: "aca8cce9-d587-4466-b46d-3fa8bc0a8583"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.362990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362807    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.362990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362827    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf21814-f278-425e-998e-a2448cb1fe12-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.362990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.362845    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.363392 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.363372    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-socket" pod "aca8cce9-d587-4466-b46d-3fa8bc0a8583" (UID: "aca8cce9-d587-4466-b46d-3fa8bc0a8583"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.364192 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.363630    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-credential-socket" pod "dcf21814-f278-425e-998e-a2448cb1fe12" (UID: "dcf21814-f278-425e-998e-a2448cb1fe12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.364336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.363800    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-data" pod "aca8cce9-d587-4466-b46d-3fa8bc0a8583" (UID: "aca8cce9-d587-4466-b46d-3fa8bc0a8583"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.364336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.363930    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-credential-socket" pod "aca8cce9-d587-4466-b46d-3fa8bc0a8583" (UID: "aca8cce9-d587-4466-b46d-3fa8bc0a8583"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.364336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.364023    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-socket" pod "dcf21814-f278-425e-998e-a2448cb1fe12" (UID: "dcf21814-f278-425e-998e-a2448cb1fe12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.364336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.364118    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-certs" pod "dcf21814-f278-425e-998e-a2448cb1fe12" (UID: "dcf21814-f278-425e-998e-a2448cb1fe12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.364336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.364228    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istiod-ca-cert" pod "aca8cce9-d587-4466-b46d-3fa8bc0a8583" (UID: "aca8cce9-d587-4466-b46d-3fa8bc0a8583"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:24.367867 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.367821    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-kube-api-access-kp4t2" pod "aca8cce9-d587-4466-b46d-3fa8bc0a8583" (UID: "aca8cce9-d587-4466-b46d-3fa8bc0a8583"). InnerVolumeSpecName "kube-api-access-kp4t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:24.368071 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.368036    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcf21814-f278-425e-998e-a2448cb1fe12-istio-podinfo" pod "dcf21814-f278-425e-998e-a2448cb1fe12" (UID: "dcf21814-f278-425e-998e-a2448cb1fe12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:24.368138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.368075    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-kube-api-access-2r74f" pod "dcf21814-f278-425e-998e-a2448cb1fe12" (UID: "dcf21814-f278-425e-998e-a2448cb1fe12"). InnerVolumeSpecName "kube-api-access-2r74f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:24.368332 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.368304    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-envoy" pod "aca8cce9-d587-4466-b46d-3fa8bc0a8583" (UID: "aca8cce9-d587-4466-b46d-3fa8bc0a8583"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.368400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.368303    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-podinfo" pod "aca8cce9-d587-4466-b46d-3fa8bc0a8583" (UID: "aca8cce9-d587-4466-b46d-3fa8bc0a8583"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:24.368400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.368384    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-envoy" pod "dcf21814-f278-425e-998e-a2448cb1fe12" (UID: "dcf21814-f278-425e-998e-a2448cb1fe12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.369939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.369901    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-istio-token" pod "dcf21814-f278-425e-998e-a2448cb1fe12" (UID: "dcf21814-f278-425e-998e-a2448cb1fe12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:24.370560 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.370540    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-token" pod "aca8cce9-d587-4466-b46d-3fa8bc0a8583" (UID: "aca8cce9-d587-4466-b46d-3fa8bc0a8583"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:24.464353 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464313    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464353 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464355    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf21814-f278-425e-998e-a2448cb1fe12-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464370    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464382    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464395    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464407    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464419    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464447    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r74f\" (UniqueName: \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-kube-api-access-2r74f\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464461    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf21814-f278-425e-998e-a2448cb1fe12-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464472    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp4t2\" (UniqueName: \"kubernetes.io/projected/aca8cce9-d587-4466-b46d-3fa8bc0a8583-kube-api-access-kp4t2\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464484    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464496    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464508    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8cce9-d587-4466-b46d-3fa8bc0a8583-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464523    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.464631 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.464535    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf21814-f278-425e-998e-a2448cb1fe12-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.651265 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.651234    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"]
Nov 03 20:27:24.659211 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.659155    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-lsdd7"]
Nov 03 20:27:24.670412 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.670382    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"]
Nov 03 20:27:24.675166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:24.675136    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-42z8m"]
Nov 03 20:27:25.616321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:25.616286    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" path="/var/lib/kubelet/pods/aca8cce9-d587-4466-b46d-3fa8bc0a8583/volumes"
Nov 03 20:27:25.616772 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:25.616757    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" path="/var/lib/kubelet/pods/dcf21814-f278-425e-998e-a2448cb1fe12/volumes"
Nov 03 20:27:26.554260 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554224    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"]
Nov 03 20:27:26.554632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554619    5600 cpu_manager.go:414] "containerMap: removing container" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy"
Nov 03 20:27:26.554690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554633    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy"
Nov 03 20:27:26.554690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554655    5600 cpu_manager.go:414] "containerMap: removing container" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy"
Nov 03 20:27:26.554690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554660    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy"
Nov 03 20:27:26.554690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554669    5600 cpu_manager.go:414] "containerMap: removing container" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy"
Nov 03 20:27:26.554690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554675    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy"
Nov 03 20:27:26.554690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554684    5600 cpu_manager.go:414] "containerMap: removing container" podUID="63df7cb9-548d-40c6-ab66-db4bb639501f" containerName="maas-api"
Nov 03 20:27:26.554690 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554689    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63df7cb9-548d-40c6-ab66-db4bb639501f" containerName="maas-api"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554695    5600 cpu_manager.go:414] "containerMap: removing container" podUID="0c16de55-ec16-4a5d-bb99-8793dd848e5b" containerName="revoke-keys"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554701    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c16de55-ec16-4a5d-bb99-8793dd848e5b" containerName="revoke-keys"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554719    5600 cpu_manager.go:414] "containerMap: removing container" podUID="d8868e28-6951-4711-88ba-498cf360042e" containerName="maas-api"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554724    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8868e28-6951-4711-88ba-498cf360042e" containerName="maas-api"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554731    5600 cpu_manager.go:414] "containerMap: removing container" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554735    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554743    5600 cpu_manager.go:414] "containerMap: removing container" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerName="istio-proxy"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554747    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerName="istio-proxy"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554754    5600 cpu_manager.go:414] "containerMap: removing container" podUID="da904e7b-25ba-4397-9841-61cccacfc4ae" containerName="istio-proxy"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554759    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da904e7b-25ba-4397-9841-61cccacfc4ae" containerName="istio-proxy"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554766    5600 cpu_manager.go:414] "containerMap: removing container" podUID="4683c83d-c95b-4878-9802-81982c339032" containerName="payload-processing"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554770    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4683c83d-c95b-4878-9802-81982c339032" containerName="payload-processing"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554778    5600 cpu_manager.go:414] "containerMap: removing container" podUID="69033edd-f107-4d62-874e-fb27cbe03b61" containerName="revoke-keys"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554782    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69033edd-f107-4d62-874e-fb27cbe03b61" containerName="revoke-keys"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554833    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8868e28-6951-4711-88ba-498cf360042e" containerName="maas-api"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554841    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe85ef7f-2ad1-4707-8a1e-00075b3e3f34" containerName="main"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554847    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca8cce9-d587-4466-b46d-3fa8bc0a8583" containerName="istio-proxy"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554855    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="da904e7b-25ba-4397-9841-61cccacfc4ae" containerName="istio-proxy"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554863    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="4683c83d-c95b-4878-9802-81982c339032" containerName="payload-processing"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554871    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554879    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="69033edd-f107-4d62-874e-fb27cbe03b61" containerName="revoke-keys"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554888    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1e0711-ae1b-4ab2-9be7-f391f55e5a8d" containerName="istio-proxy"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554899    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf21814-f278-425e-998e-a2448cb1fe12" containerName="istio-proxy"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554906    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy"
Nov 03 20:27:26.554908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554915    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="63df7cb9-548d-40c6-ab66-db4bb639501f" containerName="maas-api"
Nov 03 20:27:26.555951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.554922    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c16de55-ec16-4a5d-bb99-8793dd848e5b" containerName="revoke-keys"
Nov 03 20:27:26.559893 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.559869    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:26.574272 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.574245    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"]
Nov 03 20:27:26.657397 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.657363    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:27:26.657796 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.657409    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:27:26.685217 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.685177    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxkn\" (UniqueName: \"kubernetes.io/projected/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-kube-api-access-kzxkn\") pod \"maas-api-revoke-sub-9857b867-wdcz5\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:26.685711 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.685680    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9857b867-wdcz5\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:26.685849 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.685750    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9857b867-wdcz5\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:26.787710 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.787650    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9857b867-wdcz5\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:26.787924 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.787723    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9857b867-wdcz5\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:26.787924 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.787845    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxkn\" (UniqueName: \"kubernetes.io/projected/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-kube-api-access-kzxkn\") pod \"maas-api-revoke-sub-9857b867-wdcz5\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:26.788675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.788642    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9857b867-wdcz5\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:26.790397 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.790369    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9857b867-wdcz5\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:26.797033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.797006    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxkn\" (UniqueName: \"kubernetes.io/projected/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-kube-api-access-kzxkn\") pod \"maas-api-revoke-sub-9857b867-wdcz5\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:26.877790 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:26.877702    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:27.157521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:27.157495    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"]
Nov 03 20:27:27.159810 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:27:27.159786    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5ded5b_4229_4c8a_bd2f_b6642dc7eac6.slice/crio-9658538ec33d3f464b27ed213706813a6f87d329e7ef19e7f6cc7ae103e071ce WatchSource:0}: Error finding container 9658538ec33d3f464b27ed213706813a6f87d329e7ef19e7f6cc7ae103e071ce: Status 404 returned error can't find the container with id 9658538ec33d3f464b27ed213706813a6f87d329e7ef19e7f6cc7ae103e071ce
Nov 03 20:27:27.331178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:27.331136    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5" event={"ID":"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6","Type":"ContainerStarted","Data":"9658538ec33d3f464b27ed213706813a6f87d329e7ef19e7f6cc7ae103e071ce"}
Nov 03 20:27:28.347505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:28.343968    5600 generic.go:356] "Generic (PLEG): container finished" podID="5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6" containerID="c44b80c2a8d75ebf45132fdb5e836d9975e22bae7cc601e16ef91864acd28c32" exitCode=0
Nov 03 20:27:28.347505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:28.344164    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5" event={"ID":"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6","Type":"ContainerDied","Data":"c44b80c2a8d75ebf45132fdb5e836d9975e22bae7cc601e16ef91864acd28c32"}
Nov 03 20:27:28.795533 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:27:28.795495    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:27:29.495549 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:29.495521    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:29.617586 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:29.617487    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-openshift-ca\") pod \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") "
Nov 03 20:27:29.617586 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:29.617543    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-xks-ca\") pod \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") "
Nov 03 20:27:29.617830 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:29.617658    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-kube-api-access-kzxkn\" (UniqueName: \"kubernetes.io/projected/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-kube-api-access-kzxkn\") pod \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\" (UID: \"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6\") "
Nov 03 20:27:29.617942 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:29.617906    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-openshift-ca" pod "5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6" (UID: "5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:29.619976 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:29.619948    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-xks-ca" pod "5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6" (UID: "5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:29.620065 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:29.620015    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-kube-api-access-kzxkn" pod "5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6" (UID: "5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6"). InnerVolumeSpecName "kube-api-access-kzxkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:29.718938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:29.718902    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzxkn\" (UniqueName: \"kubernetes.io/projected/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-kube-api-access-kzxkn\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:29.718938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:29.718930    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:29.718938 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:29.718940    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:30.358504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:30.358475    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"
Nov 03 20:27:30.358705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:30.358470    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5" event={"ID":"5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6","Type":"ContainerDied","Data":"9658538ec33d3f464b27ed213706813a6f87d329e7ef19e7f6cc7ae103e071ce"}
Nov 03 20:27:30.358705 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:30.358597    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9658538ec33d3f464b27ed213706813a6f87d329e7ef19e7f6cc7ae103e071ce"
Nov 03 20:27:48.795571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:48.795530    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"]
Nov 03 20:27:48.795963 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:48.795796    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" containerName="istio-proxy" containerID="cri-o://0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a" gracePeriod=30
Nov 03 20:27:54.145082 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.144726    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:27:54.261042 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261005    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-kube-api-access-4bqf5\" (UniqueName: \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-kube-api-access-4bqf5\") pod \"a02b908f-874f-437d-98c9-99032b73a1f5\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") "
Nov 03 20:27:54.261223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261053    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-envoy\") pod \"a02b908f-874f-437d-98c9-99032b73a1f5\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") "
Nov 03 20:27:54.261223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261108    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-socket\") pod \"a02b908f-874f-437d-98c9-99032b73a1f5\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") "
Nov 03 20:27:54.261223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261140    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-certs\") pod \"a02b908f-874f-437d-98c9-99032b73a1f5\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") "
Nov 03 20:27:54.261223 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261180    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-data\") pod \"a02b908f-874f-437d-98c9-99032b73a1f5\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") "
Nov 03 20:27:54.261407 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261226    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-istio-token\" (UniqueName: \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-istio-token\") pod \"a02b908f-874f-437d-98c9-99032b73a1f5\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") "
Nov 03 20:27:54.261407 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261257    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a02b908f-874f-437d-98c9-99032b73a1f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02b908f-874f-437d-98c9-99032b73a1f5-istio-podinfo\") pod \"a02b908f-874f-437d-98c9-99032b73a1f5\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") "
Nov 03 20:27:54.261407 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261365    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-credential-socket\") pod \"a02b908f-874f-437d-98c9-99032b73a1f5\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") "
Nov 03 20:27:54.262106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261418    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02b908f-874f-437d-98c9-99032b73a1f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02b908f-874f-437d-98c9-99032b73a1f5-istiod-ca-cert\") pod \"a02b908f-874f-437d-98c9-99032b73a1f5\" (UID: \"a02b908f-874f-437d-98c9-99032b73a1f5\") "
Nov 03 20:27:54.262106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261509    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-certs" pod "a02b908f-874f-437d-98c9-99032b73a1f5" (UID: "a02b908f-874f-437d-98c9-99032b73a1f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:54.262106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261533    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-socket" pod "a02b908f-874f-437d-98c9-99032b73a1f5" (UID: "a02b908f-874f-437d-98c9-99032b73a1f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:54.262106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261563    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-data" pod "a02b908f-874f-437d-98c9-99032b73a1f5" (UID: "a02b908f-874f-437d-98c9-99032b73a1f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:54.262106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261770    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-credential-socket" pod "a02b908f-874f-437d-98c9-99032b73a1f5" (UID: "a02b908f-874f-437d-98c9-99032b73a1f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:54.262106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261824    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.262106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261846    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.262106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261862    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.262106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.261903    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02b908f-874f-437d-98c9-99032b73a1f5-istiod-ca-cert" pod "a02b908f-874f-437d-98c9-99032b73a1f5" (UID: "a02b908f-874f-437d-98c9-99032b73a1f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:54.263563 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.263535    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-kube-api-access-4bqf5" pod "a02b908f-874f-437d-98c9-99032b73a1f5" (UID: "a02b908f-874f-437d-98c9-99032b73a1f5"). InnerVolumeSpecName "kube-api-access-4bqf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:54.263669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.263614    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a02b908f-874f-437d-98c9-99032b73a1f5-istio-podinfo" pod "a02b908f-874f-437d-98c9-99032b73a1f5" (UID: "a02b908f-874f-437d-98c9-99032b73a1f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:54.264032 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.264016    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-envoy" pod "a02b908f-874f-437d-98c9-99032b73a1f5" (UID: "a02b908f-874f-437d-98c9-99032b73a1f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:54.264076 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.264037    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-istio-token" pod "a02b908f-874f-437d-98c9-99032b73a1f5" (UID: "a02b908f-874f-437d-98c9-99032b73a1f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:54.362877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.362774    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02b908f-874f-437d-98c9-99032b73a1f5-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.362877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.362812    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bqf5\" (UniqueName: \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-kube-api-access-4bqf5\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.362877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.362823    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.362877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.362832    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02b908f-874f-437d-98c9-99032b73a1f5-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.362877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.362841    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02b908f-874f-437d-98c9-99032b73a1f5-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.362877 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.362849    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02b908f-874f-437d-98c9-99032b73a1f5-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.503231 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.503186    5600 generic.go:356] "Generic (PLEG): container finished" podID="a02b908f-874f-437d-98c9-99032b73a1f5" containerID="0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a" exitCode=0
Nov 03 20:27:54.503487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.503266    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" event={"ID":"a02b908f-874f-437d-98c9-99032b73a1f5","Type":"ContainerDied","Data":"0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a"}
Nov 03 20:27:54.503487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.503321    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp" event={"ID":"a02b908f-874f-437d-98c9-99032b73a1f5","Type":"ContainerDied","Data":"90ca2996b4f5fb56dce21477c5319b5f875a5d4148e92f3c165134f864421888"}
Nov 03 20:27:54.503487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.503352    5600 scope.go:122] "RemoveContainer" containerID="0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a"
Nov 03 20:27:54.503719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.503561    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"
Nov 03 20:27:54.545846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.544798    5600 scope.go:122] "RemoveContainer" containerID="0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a"
Nov 03 20:27:54.545846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.544946    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"]
Nov 03 20:27:54.545846 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:27:54.545323    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a\": container with ID starting with 0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a not found: ID does not exist" containerID="0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a"
Nov 03 20:27:54.545846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.545358    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a"} err="failed to get container status \"0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a\": rpc error: code = NotFound desc = could not find container \"0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a\": container with ID starting with 0bb1f39e981ba46b57622b14e80119e4b6848090c7f16fa9aec6c2ac4758b86a not found: ID does not exist"
Nov 03 20:27:54.546604 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:54.546575    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-jrvcp"]
Nov 03 20:27:55.615627 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:27:55.615594    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" path="/var/lib/kubelet/pods/a02b908f-874f-437d-98c9-99032b73a1f5/volumes"
Nov 03 20:27:58.796189 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:27:58.796148    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:28:17.823004 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.822966    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"]
Nov 03 20:28:17.823408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.823383    5600 cpu_manager.go:414] "containerMap: removing container" podUID="5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6" containerName="revoke-keys"
Nov 03 20:28:17.823408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.823396    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6" containerName="revoke-keys"
Nov 03 20:28:17.823584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.823422    5600 cpu_manager.go:414] "containerMap: removing container" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy"
Nov 03 20:28:17.823584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.823427    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0e925b-4187-4596-8098-6a660bdb13a8" containerName="istio-proxy"
Nov 03 20:28:17.823584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.823458    5600 cpu_manager.go:414] "containerMap: removing container" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" containerName="istio-proxy"
Nov 03 20:28:17.823584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.823466    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" containerName="istio-proxy"
Nov 03 20:28:17.823584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.823561    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02b908f-874f-437d-98c9-99032b73a1f5" containerName="istio-proxy"
Nov 03 20:28:17.823584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.823576    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6" containerName="revoke-keys"
Nov 03 20:28:17.827461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.827406    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:17.841257 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.841226    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"]
Nov 03 20:28:17.994949 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.994914    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:17.994949 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.994955    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:17.995177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.994979    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:17.995177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.995051    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bcx\" (UniqueName: \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-kube-api-access-b2bcx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:17.995177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.995090    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:17.995177 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.995159    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:17.995350 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.995211    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:17.995350 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.995273    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:17.995350 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:17.995312    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.096900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.096809    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.096900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.096853    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.096900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.096869    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bcx\" (UniqueName: \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-kube-api-access-b2bcx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.096900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.096895    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.097214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.096953    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.097214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.096986    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.097370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.097316    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.097370 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.097355    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.097611 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.097420    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.097611 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.097526    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.097611 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.097532    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.097948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.097627    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.097948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.097787    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.097948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.097880    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.099522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.099501    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.099619 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.099579    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.104903 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.104881    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bcx\" (UniqueName: \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-kube-api-access-b2bcx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.104998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.104943    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vbhfd\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.150660 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.150622    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:18.430161 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.430130    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"]
Nov 03 20:28:18.432477 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:28:18.432445    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c3aae1_9313_4ddd_a223_c6feefd0eaf6.slice/crio-c4a130556fc0e3e61fdf4603a8fa0c7443a556a75979f0424a15a30440a07f37 WatchSource:0}: Error finding container c4a130556fc0e3e61fdf4603a8fa0c7443a556a75979f0424a15a30440a07f37: Status 404 returned error can't find the container with id c4a130556fc0e3e61fdf4603a8fa0c7443a556a75979f0424a15a30440a07f37
Nov 03 20:28:18.434607 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.434573    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:28:18.434713 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.434675    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:28:18.434774 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.434720    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:28:18.634514 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.634476    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" event={"ID":"61c3aae1-9313-4ddd-a223-c6feefd0eaf6","Type":"ContainerStarted","Data":"4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4"}
Nov 03 20:28:18.634688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.634532    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" event={"ID":"61c3aae1-9313-4ddd-a223-c6feefd0eaf6","Type":"ContainerStarted","Data":"c4a130556fc0e3e61fdf4603a8fa0c7443a556a75979f0424a15a30440a07f37"}
Nov 03 20:28:18.654295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:18.654236    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" podStartSLOduration=1.654217591 podStartE2EDuration="1.654217591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:18.652516893 +0000 UTC m=+2881.165338676" watchObservedRunningTime="2026-11-03 20:28:18.654217591 +0000 UTC m=+2881.167039374"
Nov 03 20:28:19.151104 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:19.151063    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:19.152632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:19.152604    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vbhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:28:19.152764 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:19.152671    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:28:20.151836 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:20.151800    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vbhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:28:20.152279 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:20.151852    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:28:21.151712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:21.151666    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vbhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:28:21.151905 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:21.151732    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:28:22.151274 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:22.151238    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vbhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:28:22.151485 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:22.151298    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:28:23.151865 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:23.151827    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vbhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:28:23.152320 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:23.151883    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:28:24.151519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:24.151476    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vbhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:28:24.151758 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:24.151561    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:28:25.151717 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:25.151678    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vbhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:28:25.152215 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:25.151730    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:28:26.676098 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:26.675892    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:26.676596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:26.676356    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:26.677618 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:28:26.677590    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:28:28.795184 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:28:28.795145    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:28:58.794660 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:28:58.794615    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:29:28.794889 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:29:28.794851    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:29:58.794292 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:29:58.794208    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:30:05.541504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.541451    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"]
Nov 03 20:30:05.545426 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.545401    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:05.550991 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.550970    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:30:05.552088 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.552066    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:30:05.554834 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.554811    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"]
Nov 03 20:30:05.566821 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.566795    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n76hf\" (UniqueName: \"kubernetes.io/projected/96b2f190-16d9-49c1-bc2e-5718c31450aa-kube-api-access-n76hf\") pod \"maas-api-revoke-sub-046e1343-2n858\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:05.566959 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.566848    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-2n858\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:05.567035 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.566971    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-2n858\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:05.667423 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.667386    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76hf\" (UniqueName: \"kubernetes.io/projected/96b2f190-16d9-49c1-bc2e-5718c31450aa-kube-api-access-n76hf\") pod \"maas-api-revoke-sub-046e1343-2n858\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:05.667658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.667473    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-2n858\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:05.667658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.667571    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-2n858\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:05.668264 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.668241    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-2n858\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:05.669987 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.669965    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-2n858\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:05.675388 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.675365    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n76hf\" (UniqueName: \"kubernetes.io/projected/96b2f190-16d9-49c1-bc2e-5718c31450aa-kube-api-access-n76hf\") pod \"maas-api-revoke-sub-046e1343-2n858\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:05.860693 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:05.860606    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:06.136924 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:06.136892    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"]
Nov 03 20:30:06.139076 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:30:06.139049    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b2f190_16d9_49c1_bc2e_5718c31450aa.slice/crio-0678565050e2feaa1106c40779804ee157ffc01594a642f61db086c9dd0eea2b WatchSource:0}: Error finding container 0678565050e2feaa1106c40779804ee157ffc01594a642f61db086c9dd0eea2b: Status 404 returned error can't find the container with id 0678565050e2feaa1106c40779804ee157ffc01594a642f61db086c9dd0eea2b
Nov 03 20:30:06.299817 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:06.299779    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858" event={"ID":"96b2f190-16d9-49c1-bc2e-5718c31450aa","Type":"ContainerStarted","Data":"0678565050e2feaa1106c40779804ee157ffc01594a642f61db086c9dd0eea2b"}
Nov 03 20:30:07.306538 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:07.306428    5600 generic.go:356] "Generic (PLEG): container finished" podID="96b2f190-16d9-49c1-bc2e-5718c31450aa" containerID="97c561edf0c84949a6dd424500e2c96a8577b4ae158231f177635fd834191753" exitCode=0
Nov 03 20:30:07.306944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:07.306534    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858" event={"ID":"96b2f190-16d9-49c1-bc2e-5718c31450aa","Type":"ContainerDied","Data":"97c561edf0c84949a6dd424500e2c96a8577b4ae158231f177635fd834191753"}
Nov 03 20:30:08.443728 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:08.443705    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:08.494746 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:08.494717    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b2f190-16d9-49c1-bc2e-5718c31450aa-kube-api-access-n76hf\" (UniqueName: \"kubernetes.io/projected/96b2f190-16d9-49c1-bc2e-5718c31450aa-kube-api-access-n76hf\") pod \"96b2f190-16d9-49c1-bc2e-5718c31450aa\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") "
Nov 03 20:30:08.494940 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:08.494767    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-xks-ca\") pod \"96b2f190-16d9-49c1-bc2e-5718c31450aa\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") "
Nov 03 20:30:08.495047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:08.494947    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-openshift-ca\") pod \"96b2f190-16d9-49c1-bc2e-5718c31450aa\" (UID: \"96b2f190-16d9-49c1-bc2e-5718c31450aa\") "
Nov 03 20:30:08.495325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:08.495287    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-openshift-ca" pod "96b2f190-16d9-49c1-bc2e-5718c31450aa" (UID: "96b2f190-16d9-49c1-bc2e-5718c31450aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:08.496996 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:08.496972    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-xks-ca" pod "96b2f190-16d9-49c1-bc2e-5718c31450aa" (UID: "96b2f190-16d9-49c1-bc2e-5718c31450aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:08.497106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:08.497043    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b2f190-16d9-49c1-bc2e-5718c31450aa-kube-api-access-n76hf" pod "96b2f190-16d9-49c1-bc2e-5718c31450aa" (UID: "96b2f190-16d9-49c1-bc2e-5718c31450aa"). InnerVolumeSpecName "kube-api-access-n76hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:08.596777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:08.596694    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n76hf\" (UniqueName: \"kubernetes.io/projected/96b2f190-16d9-49c1-bc2e-5718c31450aa-kube-api-access-n76hf\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:08.596777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:08.596726    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:08.596777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:08.596735    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b2f190-16d9-49c1-bc2e-5718c31450aa-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:09.318186 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:09.318155    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"
Nov 03 20:30:09.318375 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:09.318202    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858" event={"ID":"96b2f190-16d9-49c1-bc2e-5718c31450aa","Type":"ContainerDied","Data":"0678565050e2feaa1106c40779804ee157ffc01594a642f61db086c9dd0eea2b"}
Nov 03 20:30:09.318375 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:09.318252    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0678565050e2feaa1106c40779804ee157ffc01594a642f61db086c9dd0eea2b"
Nov 03 20:30:15.801993 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.801952    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"]
Nov 03 20:30:15.802489 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.802353    5600 cpu_manager.go:414] "containerMap: removing container" podUID="96b2f190-16d9-49c1-bc2e-5718c31450aa" containerName="revoke-keys"
Nov 03 20:30:15.802489 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.802366    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b2f190-16d9-49c1-bc2e-5718c31450aa" containerName="revoke-keys"
Nov 03 20:30:15.802489 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.802488    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b2f190-16d9-49c1-bc2e-5718c31450aa" containerName="revoke-keys"
Nov 03 20:30:15.805629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.805613    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:15.813931 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.813908    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"]
Nov 03 20:30:15.818682 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.818651    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:30:15.818831 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.818758    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:30:15.865918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.865875    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-p52xw\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:15.866091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.865924    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-p52xw\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:15.866091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.866052    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvnc\" (UniqueName: \"kubernetes.io/projected/cd9a642d-b921-4d96-a54c-0ce3164651b6-kube-api-access-zlvnc\") pod \"maas-api-revoke-sub-d3fba056-p52xw\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:15.967137 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.967100    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-p52xw\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:15.967327 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.967178    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvnc\" (UniqueName: \"kubernetes.io/projected/cd9a642d-b921-4d96-a54c-0ce3164651b6-kube-api-access-zlvnc\") pod \"maas-api-revoke-sub-d3fba056-p52xw\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:15.967327 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.967251    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-p52xw\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:15.967974 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.967953    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-p52xw\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:15.969828 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.969796    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-p52xw\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:15.975704 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:15.975686    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvnc\" (UniqueName: \"kubernetes.io/projected/cd9a642d-b921-4d96-a54c-0ce3164651b6-kube-api-access-zlvnc\") pod \"maas-api-revoke-sub-d3fba056-p52xw\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:16.120355 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:16.120267    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:16.396509 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:16.396407    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"]
Nov 03 20:30:16.399159 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:30:16.399127    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9a642d_b921_4d96_a54c_0ce3164651b6.slice/crio-38ddfd7273dc8481515bda34e22f987d253fa86061dc33764e603e091b78ef3b WatchSource:0}: Error finding container 38ddfd7273dc8481515bda34e22f987d253fa86061dc33764e603e091b78ef3b: Status 404 returned error can't find the container with id 38ddfd7273dc8481515bda34e22f987d253fa86061dc33764e603e091b78ef3b
Nov 03 20:30:17.363378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:17.363339    5600 generic.go:356] "Generic (PLEG): container finished" podID="cd9a642d-b921-4d96-a54c-0ce3164651b6" containerID="f2780ce30783555ffbb5096cee9564ad37ee8ea270277fc5b86e84ea35991cc2" exitCode=0
Nov 03 20:30:17.363954 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:17.363899    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw" event={"ID":"cd9a642d-b921-4d96-a54c-0ce3164651b6","Type":"ContainerDied","Data":"f2780ce30783555ffbb5096cee9564ad37ee8ea270277fc5b86e84ea35991cc2"}
Nov 03 20:30:17.364045 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:17.363957    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw" event={"ID":"cd9a642d-b921-4d96-a54c-0ce3164651b6","Type":"ContainerStarted","Data":"38ddfd7273dc8481515bda34e22f987d253fa86061dc33764e603e091b78ef3b"}
Nov 03 20:30:18.865059 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:18.864693    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:18.929465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:18.917769    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9a642d-b921-4d96-a54c-0ce3164651b6-kube-api-access-zlvnc\" (UniqueName: \"kubernetes.io/projected/cd9a642d-b921-4d96-a54c-0ce3164651b6-kube-api-access-zlvnc\") pod \"cd9a642d-b921-4d96-a54c-0ce3164651b6\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") "
Nov 03 20:30:18.929465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:18.917821    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-openshift-ca\") pod \"cd9a642d-b921-4d96-a54c-0ce3164651b6\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") "
Nov 03 20:30:18.929465 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:18.917978    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-xks-ca\") pod \"cd9a642d-b921-4d96-a54c-0ce3164651b6\" (UID: \"cd9a642d-b921-4d96-a54c-0ce3164651b6\") "
Nov 03 20:30:18.960741 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:18.960655    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-openshift-ca" pod "cd9a642d-b921-4d96-a54c-0ce3164651b6" (UID: "cd9a642d-b921-4d96-a54c-0ce3164651b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:18.982469 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:18.980944    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-xks-ca" pod "cd9a642d-b921-4d96-a54c-0ce3164651b6" (UID: "cd9a642d-b921-4d96-a54c-0ce3164651b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:18.982806 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:18.982760    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9a642d-b921-4d96-a54c-0ce3164651b6-kube-api-access-zlvnc" pod "cd9a642d-b921-4d96-a54c-0ce3164651b6" (UID: "cd9a642d-b921-4d96-a54c-0ce3164651b6"). InnerVolumeSpecName "kube-api-access-zlvnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:19.047519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:19.024100    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlvnc\" (UniqueName: \"kubernetes.io/projected/cd9a642d-b921-4d96-a54c-0ce3164651b6-kube-api-access-zlvnc\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:19.047519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:19.024141    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:19.047519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:19.024160    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd9a642d-b921-4d96-a54c-0ce3164651b6-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:19.379875 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:19.379781    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"
Nov 03 20:30:19.379875 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:19.379813    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw" event={"ID":"cd9a642d-b921-4d96-a54c-0ce3164651b6","Type":"ContainerDied","Data":"38ddfd7273dc8481515bda34e22f987d253fa86061dc33764e603e091b78ef3b"}
Nov 03 20:30:19.380104 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:30:19.379887    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38ddfd7273dc8481515bda34e22f987d253fa86061dc33764e603e091b78ef3b"
Nov 03 20:30:28.797016 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:30:28.796973    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:30:58.795873 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:30:58.795831    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:31:27.876011 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:27.875903    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"]
Nov 03 20:31:27.876693 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:27.876666    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z" podUID="5e010c68-f9f3-48c9-b578-d61dbbbf92eb" containerName="discovery" containerID="cri-o://213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08" gracePeriod=30
Nov 03 20:31:27.906464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:27.898543    5600 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qrs7z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:8080/ready\": dial tcp 10.134.0.85:8080: connect: connection refused" start-of-body=
Nov 03 20:31:27.906464 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:27.898619    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z" podUID="5e010c68-f9f3-48c9-b578-d61dbbbf92eb" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.85:8080/ready\": dial tcp 10.134.0.85:8080: connect: connection refused"
Nov 03 20:31:28.464504 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.462758    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:31:28.597061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.575715    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-token\" (UniqueName: \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-token\") pod \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") "
Nov 03 20:31:28.597061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.575795    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-local-certs\") pod \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") "
Nov 03 20:31:28.597061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.575827    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-cacerts\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-cacerts\") pod \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") "
Nov 03 20:31:28.597061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.575870    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-ca-configmap\") pod \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") "
Nov 03 20:31:28.597061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.575976    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-dns-cert\") pod \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") "
Nov 03 20:31:28.597061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.576027    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-kubeconfig\") pod \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") "
Nov 03 20:31:28.597061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.576084    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-kube-api-access-47wf4\" (UniqueName: \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-kube-api-access-47wf4\") pod \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\" (UID: \"5e010c68-f9f3-48c9-b578-d61dbbbf92eb\") "
Nov 03 20:31:28.597061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.580718    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-ca-configmap" pod "5e010c68-f9f3-48c9-b578-d61dbbbf92eb" (UID: "5e010c68-f9f3-48c9-b578-d61dbbbf92eb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:28.660326 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.660265    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-cacerts" pod "5e010c68-f9f3-48c9-b578-d61dbbbf92eb" (UID: "5e010c68-f9f3-48c9-b578-d61dbbbf92eb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:28.661042 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.661000    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-kubeconfig" pod "5e010c68-f9f3-48c9-b578-d61dbbbf92eb" (UID: "5e010c68-f9f3-48c9-b578-d61dbbbf92eb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:28.661346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.661308    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-token" pod "5e010c68-f9f3-48c9-b578-d61dbbbf92eb" (UID: "5e010c68-f9f3-48c9-b578-d61dbbbf92eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:28.661650 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.661612    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-kube-api-access-47wf4" pod "5e010c68-f9f3-48c9-b578-d61dbbbf92eb" (UID: "5e010c68-f9f3-48c9-b578-d61dbbbf92eb"). InnerVolumeSpecName "kube-api-access-47wf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:28.663909 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.663874    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-dns-cert" pod "5e010c68-f9f3-48c9-b578-d61dbbbf92eb" (UID: "5e010c68-f9f3-48c9-b578-d61dbbbf92eb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:28.665584 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.665550    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-local-certs" pod "5e010c68-f9f3-48c9-b578-d61dbbbf92eb" (UID: "5e010c68-f9f3-48c9-b578-d61dbbbf92eb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:28.677180 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.677150    5600 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-ca-configmap\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:28.677180 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.677173    5600 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-csr-dns-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:28.677324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.677187    5600 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-kubeconfig\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:28.677324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.677202    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47wf4\" (UniqueName: \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-kube-api-access-47wf4\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:28.677324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.677214    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:28.677324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.677228    5600 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-local-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:28.677324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.677241    5600 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e010c68-f9f3-48c9-b578-d61dbbbf92eb-cacerts\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:28.808128 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:31:28.808025    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:31:28.853253 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.853215    5600 generic.go:356] "Generic (PLEG): container finished" podID="5e010c68-f9f3-48c9-b578-d61dbbbf92eb" containerID="213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08" exitCode=0
Nov 03 20:31:28.853473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.853270    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"
Nov 03 20:31:28.853473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.853309    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z" event={"ID":"5e010c68-f9f3-48c9-b578-d61dbbbf92eb","Type":"ContainerDied","Data":"213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08"}
Nov 03 20:31:28.853473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.853370    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z" event={"ID":"5e010c68-f9f3-48c9-b578-d61dbbbf92eb","Type":"ContainerDied","Data":"c1448b4ba6800a3d8322b25bb5d64c98cdb5caa15cef5b1bbac79e708dc6e143"}
Nov 03 20:31:28.853473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.853392    5600 scope.go:122] "RemoveContainer" containerID="213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08"
Nov 03 20:31:28.864419 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.864400    5600 scope.go:122] "RemoveContainer" containerID="213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08"
Nov 03 20:31:28.864747 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:31:28.864727    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08\": container with ID starting with 213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08 not found: ID does not exist" containerID="213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08"
Nov 03 20:31:28.864833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.864755    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08"} err="failed to get container status \"213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08\": rpc error: code = NotFound desc = could not find container \"213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08\": container with ID starting with 213bf60ce24b488aebd284bc456639191b2784258e74d365aaea93cf614d1f08 not found: ID does not exist"
Nov 03 20:31:28.875243 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.875213    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"]
Nov 03 20:31:28.880134 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:28.880112    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrs7z"]
Nov 03 20:31:29.618037 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:29.618003    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e010c68-f9f3-48c9-b578-d61dbbbf92eb" path="/var/lib/kubelet/pods/5e010c68-f9f3-48c9-b578-d61dbbbf92eb/volumes"
Nov 03 20:31:30.927049 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.927018    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"]
Nov 03 20:31:30.927526 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.927509    5600 cpu_manager.go:414] "containerMap: removing container" podUID="5e010c68-f9f3-48c9-b578-d61dbbbf92eb" containerName="discovery"
Nov 03 20:31:30.927600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.927530    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e010c68-f9f3-48c9-b578-d61dbbbf92eb" containerName="discovery"
Nov 03 20:31:30.927600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.927546    5600 cpu_manager.go:414] "containerMap: removing container" podUID="cd9a642d-b921-4d96-a54c-0ce3164651b6" containerName="revoke-keys"
Nov 03 20:31:30.927600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.927551    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9a642d-b921-4d96-a54c-0ce3164651b6" containerName="revoke-keys"
Nov 03 20:31:30.927712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.927632    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e010c68-f9f3-48c9-b578-d61dbbbf92eb" containerName="discovery"
Nov 03 20:31:30.927712 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.927643    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9a642d-b921-4d96-a54c-0ce3164651b6" containerName="revoke-keys"
Nov 03 20:31:30.932501 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.932481    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:30.937251 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.937227    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:31:30.937369 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.937291    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:31:30.938581 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.938544    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"]
Nov 03 20:31:30.997293 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.997260    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rl5kv\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:30.997482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.997301    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rl5kv\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:30.997482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:30.997378    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hnf\" (UniqueName: \"kubernetes.io/projected/4e6e2559-1e92-4e53-97c3-7b1273464d37-kube-api-access-48hnf\") pod \"maas-api-revoke-sub-0b7e8edb-rl5kv\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:31.098411 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:31.098379    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rl5kv\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:31.098599 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:31.098453    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rl5kv\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:31.098599 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:31.098493    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hnf\" (UniqueName: \"kubernetes.io/projected/4e6e2559-1e92-4e53-97c3-7b1273464d37-kube-api-access-48hnf\") pod \"maas-api-revoke-sub-0b7e8edb-rl5kv\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:31.099016 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:31.098995    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rl5kv\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:31.101039 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:31.101014    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rl5kv\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:31.106874 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:31.106849    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hnf\" (UniqueName: \"kubernetes.io/projected/4e6e2559-1e92-4e53-97c3-7b1273464d37-kube-api-access-48hnf\") pod \"maas-api-revoke-sub-0b7e8edb-rl5kv\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:31.248889 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:31.248801    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:31.521675 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:31.521649    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"]
Nov 03 20:31:31.523726 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:31:31.523699    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6e2559_1e92_4e53_97c3_7b1273464d37.slice/crio-0e7490273740cb9a53b17679e3213f8b097fe52a680b12df450a1365ea625ac6 WatchSource:0}: Error finding container 0e7490273740cb9a53b17679e3213f8b097fe52a680b12df450a1365ea625ac6: Status 404 returned error can't find the container with id 0e7490273740cb9a53b17679e3213f8b097fe52a680b12df450a1365ea625ac6
Nov 03 20:31:31.871473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:31.871352    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv" event={"ID":"4e6e2559-1e92-4e53-97c3-7b1273464d37","Type":"ContainerStarted","Data":"3f0522a1c38a937e600c0f0e4979bedaaa80445bfc05b436eac39e2164c42311"}
Nov 03 20:31:31.871473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:31.871402    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv" event={"ID":"4e6e2559-1e92-4e53-97c3-7b1273464d37","Type":"ContainerStarted","Data":"0e7490273740cb9a53b17679e3213f8b097fe52a680b12df450a1365ea625ac6"}
Nov 03 20:31:31.890587 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:31.890541    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv" podStartSLOduration=1.890528465 podStartE2EDuration="1.890528465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:31.886803704 +0000 UTC m=+3074.399625484" watchObservedRunningTime="2026-11-03 20:31:31.890528465 +0000 UTC m=+3074.403350247"
Nov 03 20:31:32.877855 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:32.877819    5600 generic.go:356] "Generic (PLEG): container finished" podID="4e6e2559-1e92-4e53-97c3-7b1273464d37" containerID="3f0522a1c38a937e600c0f0e4979bedaaa80445bfc05b436eac39e2164c42311" exitCode=0
Nov 03 20:31:32.878288 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:32.877906    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv" event={"ID":"4e6e2559-1e92-4e53-97c3-7b1273464d37","Type":"ContainerDied","Data":"3f0522a1c38a937e600c0f0e4979bedaaa80445bfc05b436eac39e2164c42311"}
Nov 03 20:31:34.017116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.017091    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:34.131880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.131845    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-openshift-ca\") pod \"4e6e2559-1e92-4e53-97c3-7b1273464d37\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") "
Nov 03 20:31:34.132056 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.131895    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6e2559-1e92-4e53-97c3-7b1273464d37-kube-api-access-48hnf\" (UniqueName: \"kubernetes.io/projected/4e6e2559-1e92-4e53-97c3-7b1273464d37-kube-api-access-48hnf\") pod \"4e6e2559-1e92-4e53-97c3-7b1273464d37\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") "
Nov 03 20:31:34.132056 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.131945    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-xks-ca\") pod \"4e6e2559-1e92-4e53-97c3-7b1273464d37\" (UID: \"4e6e2559-1e92-4e53-97c3-7b1273464d37\") "
Nov 03 20:31:34.132230 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.132210    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-openshift-ca" pod "4e6e2559-1e92-4e53-97c3-7b1273464d37" (UID: "4e6e2559-1e92-4e53-97c3-7b1273464d37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:34.134242 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.134179    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-xks-ca" pod "4e6e2559-1e92-4e53-97c3-7b1273464d37" (UID: "4e6e2559-1e92-4e53-97c3-7b1273464d37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:34.134372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.134357    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6e2559-1e92-4e53-97c3-7b1273464d37-kube-api-access-48hnf" pod "4e6e2559-1e92-4e53-97c3-7b1273464d37" (UID: "4e6e2559-1e92-4e53-97c3-7b1273464d37"). InnerVolumeSpecName "kube-api-access-48hnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:34.233494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.233465    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:34.233494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.233493    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e6e2559-1e92-4e53-97c3-7b1273464d37-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:34.233719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.233503    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48hnf\" (UniqueName: \"kubernetes.io/projected/4e6e2559-1e92-4e53-97c3-7b1273464d37-kube-api-access-48hnf\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:34.889831 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.889797    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv" event={"ID":"4e6e2559-1e92-4e53-97c3-7b1273464d37","Type":"ContainerDied","Data":"0e7490273740cb9a53b17679e3213f8b097fe52a680b12df450a1365ea625ac6"}
Nov 03 20:31:34.889831 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.889827    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"
Nov 03 20:31:34.890063 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:34.889846    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e7490273740cb9a53b17679e3213f8b097fe52a680b12df450a1365ea625ac6"
Nov 03 20:31:41.648091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.648058    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"]
Nov 03 20:31:41.648519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.648501    5600 cpu_manager.go:414] "containerMap: removing container" podUID="4e6e2559-1e92-4e53-97c3-7b1273464d37" containerName="revoke-keys"
Nov 03 20:31:41.648566 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.648521    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6e2559-1e92-4e53-97c3-7b1273464d37" containerName="revoke-keys"
Nov 03 20:31:41.648616 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.648605    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6e2559-1e92-4e53-97c3-7b1273464d37" containerName="revoke-keys"
Nov 03 20:31:41.652004 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.651980    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:41.668321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.668290    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:31:41.668470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.668327    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"]
Nov 03 20:31:41.668470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.668394    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:31:41.804617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.804576    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r6t4t\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:41.804827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.804654    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46qb7\" (UniqueName: \"kubernetes.io/projected/df9fff39-5a7c-4cf3-93e9-5da39eb70078-kube-api-access-46qb7\") pod \"maas-api-revoke-sub-3d4156f1-r6t4t\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:41.804827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.804760    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r6t4t\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:41.905565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.905483    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r6t4t\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:41.905565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.905531    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46qb7\" (UniqueName: \"kubernetes.io/projected/df9fff39-5a7c-4cf3-93e9-5da39eb70078-kube-api-access-46qb7\") pod \"maas-api-revoke-sub-3d4156f1-r6t4t\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:41.905565 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.905558    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r6t4t\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:41.906216 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.906189    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r6t4t\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:41.908125 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.908106    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r6t4t\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:41.914236 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.914213    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46qb7\" (UniqueName: \"kubernetes.io/projected/df9fff39-5a7c-4cf3-93e9-5da39eb70078-kube-api-access-46qb7\") pod \"maas-api-revoke-sub-3d4156f1-r6t4t\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:41.970190 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:41.970152    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:42.246460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:42.246401    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"]
Nov 03 20:31:42.249391 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:31:42.249364    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9fff39_5a7c_4cf3_93e9_5da39eb70078.slice/crio-3708e766cd5a52f9dafdd34a5c492f63233bf54d2bdb64ed40f8667bda865972 WatchSource:0}: Error finding container 3708e766cd5a52f9dafdd34a5c492f63233bf54d2bdb64ed40f8667bda865972: Status 404 returned error can't find the container with id 3708e766cd5a52f9dafdd34a5c492f63233bf54d2bdb64ed40f8667bda865972
Nov 03 20:31:42.933515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:42.933481    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t" event={"ID":"df9fff39-5a7c-4cf3-93e9-5da39eb70078","Type":"ContainerStarted","Data":"c719bb3e39493f5250aaa29a39d0dabad7419bd2aa87f873611e66c1b425d4b8"}
Nov 03 20:31:42.933929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:42.933534    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t" event={"ID":"df9fff39-5a7c-4cf3-93e9-5da39eb70078","Type":"ContainerStarted","Data":"3708e766cd5a52f9dafdd34a5c492f63233bf54d2bdb64ed40f8667bda865972"}
Nov 03 20:31:42.953887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:42.953829    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t" podStartSLOduration=1.953815592 podStartE2EDuration="1.953815592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:42.951852081 +0000 UTC m=+3085.464673862" watchObservedRunningTime="2026-11-03 20:31:42.953815592 +0000 UTC m=+3085.466637374"
Nov 03 20:31:43.949127 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:43.949087    5600 generic.go:356] "Generic (PLEG): container finished" podID="df9fff39-5a7c-4cf3-93e9-5da39eb70078" containerID="c719bb3e39493f5250aaa29a39d0dabad7419bd2aa87f873611e66c1b425d4b8" exitCode=0
Nov 03 20:31:43.949670 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:43.949170    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t" event={"ID":"df9fff39-5a7c-4cf3-93e9-5da39eb70078","Type":"ContainerDied","Data":"c719bb3e39493f5250aaa29a39d0dabad7419bd2aa87f873611e66c1b425d4b8"}
Nov 03 20:31:45.087975 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.087952    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:45.239131 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.239043    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-xks-ca\") pod \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") "
Nov 03 20:31:45.239285 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.239155    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9fff39-5a7c-4cf3-93e9-5da39eb70078-kube-api-access-46qb7\" (UniqueName: \"kubernetes.io/projected/df9fff39-5a7c-4cf3-93e9-5da39eb70078-kube-api-access-46qb7\") pod \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") "
Nov 03 20:31:45.239285 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.239179    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-openshift-ca\") pod \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\" (UID: \"df9fff39-5a7c-4cf3-93e9-5da39eb70078\") "
Nov 03 20:31:45.239648 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.239618    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-openshift-ca" pod "df9fff39-5a7c-4cf3-93e9-5da39eb70078" (UID: "df9fff39-5a7c-4cf3-93e9-5da39eb70078"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:45.241163 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.241138    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-xks-ca" pod "df9fff39-5a7c-4cf3-93e9-5da39eb70078" (UID: "df9fff39-5a7c-4cf3-93e9-5da39eb70078"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:45.241324 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.241302    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9fff39-5a7c-4cf3-93e9-5da39eb70078-kube-api-access-46qb7" pod "df9fff39-5a7c-4cf3-93e9-5da39eb70078" (UID: "df9fff39-5a7c-4cf3-93e9-5da39eb70078"). InnerVolumeSpecName "kube-api-access-46qb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:45.340199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.340166    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:45.340199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.340197    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46qb7\" (UniqueName: \"kubernetes.io/projected/df9fff39-5a7c-4cf3-93e9-5da39eb70078-kube-api-access-46qb7\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:45.340199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.340207    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df9fff39-5a7c-4cf3-93e9-5da39eb70078-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:45.961798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.961770    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"
Nov 03 20:31:45.961798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.961777    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t" event={"ID":"df9fff39-5a7c-4cf3-93e9-5da39eb70078","Type":"ContainerDied","Data":"3708e766cd5a52f9dafdd34a5c492f63233bf54d2bdb64ed40f8667bda865972"}
Nov 03 20:31:45.961999 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:45.961823    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3708e766cd5a52f9dafdd34a5c492f63233bf54d2bdb64ed40f8667bda865972"
Nov 03 20:31:57.731459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:57.730806    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vbhfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:31:57.731459 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:57.730907    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:31:58.795700 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:31:58.795656    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:31:59.679286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.679246    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"]
Nov 03 20:31:59.697522 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.697486    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r6t4t"]
Nov 03 20:31:59.733468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.733415    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"]
Nov 03 20:31:59.733842 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.733828    5600 cpu_manager.go:414] "containerMap: removing container" podUID="df9fff39-5a7c-4cf3-93e9-5da39eb70078" containerName="revoke-keys"
Nov 03 20:31:59.733936 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.733845    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9fff39-5a7c-4cf3-93e9-5da39eb70078" containerName="revoke-keys"
Nov 03 20:31:59.733936 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.733931    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9fff39-5a7c-4cf3-93e9-5da39eb70078" containerName="revoke-keys"
Nov 03 20:31:59.737114 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.737099    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:31:59.748180 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.748159    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:31:59.748354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.748333    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:31:59.750836 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.750808    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"]
Nov 03 20:31:59.854506 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.854468    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfhn2\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:31:59.854951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.854535    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nmnd\" (UniqueName: \"kubernetes.io/projected/217bd28e-15f2-42b3-8946-06fa60fcd95f-kube-api-access-9nmnd\") pod \"maas-api-revoke-sub-3d4156f1-rfhn2\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:31:59.854951 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.854677    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfhn2\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:31:59.955350 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.955262    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfhn2\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:31:59.955350 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.955310    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nmnd\" (UniqueName: \"kubernetes.io/projected/217bd28e-15f2-42b3-8946-06fa60fcd95f-kube-api-access-9nmnd\") pod \"maas-api-revoke-sub-3d4156f1-rfhn2\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:31:59.955582 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.955371    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfhn2\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:31:59.956186 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.956160    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfhn2\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:31:59.957806 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.957787    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rfhn2\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:31:59.963471 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:31:59.963419    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nmnd\" (UniqueName: \"kubernetes.io/projected/217bd28e-15f2-42b3-8946-06fa60fcd95f-kube-api-access-9nmnd\") pod \"maas-api-revoke-sub-3d4156f1-rfhn2\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:32:00.050659 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:00.050602    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:32:00.325658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:00.325633    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"]
Nov 03 20:32:00.327283 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:32:00.327248    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217bd28e_15f2_42b3_8946_06fa60fcd95f.slice/crio-79480e07d211f460438d504ed07125f5ca538c22c661c0475fcf5ea2eefe314f WatchSource:0}: Error finding container 79480e07d211f460438d504ed07125f5ca538c22c661c0475fcf5ea2eefe314f: Status 404 returned error can't find the container with id 79480e07d211f460438d504ed07125f5ca538c22c661c0475fcf5ea2eefe314f
Nov 03 20:32:01.022160 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:01.022119    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"]
Nov 03 20:32:01.027674 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:01.027644    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73c2e74-22gft"]
Nov 03 20:32:01.063878 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:01.063834    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2" event={"ID":"217bd28e-15f2-42b3-8946-06fa60fcd95f","Type":"ContainerStarted","Data":"bff9c8daf0befd3d0382155f360e124392a55b7540eac343a9fa9b644449aae4"}
Nov 03 20:32:01.064035 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:01.063901    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2" event={"ID":"217bd28e-15f2-42b3-8946-06fa60fcd95f","Type":"ContainerStarted","Data":"79480e07d211f460438d504ed07125f5ca538c22c661c0475fcf5ea2eefe314f"}
Nov 03 20:32:01.082070 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:01.082021    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2" podStartSLOduration=2.082007344 podStartE2EDuration="2.082007344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:01.0793968 +0000 UTC m=+3103.592218605" watchObservedRunningTime="2026-11-03 20:32:01.082007344 +0000 UTC m=+3103.594829125"
Nov 03 20:32:01.615192 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:01.615151    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69033edd-f107-4d62-874e-fb27cbe03b61" path="/var/lib/kubelet/pods/69033edd-f107-4d62-874e-fb27cbe03b61/volumes"
Nov 03 20:32:01.615674 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:01.615658    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9fff39-5a7c-4cf3-93e9-5da39eb70078" path="/var/lib/kubelet/pods/df9fff39-5a7c-4cf3-93e9-5da39eb70078/volumes"
Nov 03 20:32:02.071858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:02.071820    5600 generic.go:356] "Generic (PLEG): container finished" podID="217bd28e-15f2-42b3-8946-06fa60fcd95f" containerID="bff9c8daf0befd3d0382155f360e124392a55b7540eac343a9fa9b644449aae4" exitCode=0
Nov 03 20:32:02.072261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:02.071894    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2" event={"ID":"217bd28e-15f2-42b3-8946-06fa60fcd95f","Type":"ContainerDied","Data":"bff9c8daf0befd3d0382155f360e124392a55b7540eac343a9fa9b644449aae4"}
Nov 03 20:32:03.213833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:03.213805    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:32:03.283457 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:03.283407    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/217bd28e-15f2-42b3-8946-06fa60fcd95f-kube-api-access-9nmnd\" (UniqueName: \"kubernetes.io/projected/217bd28e-15f2-42b3-8946-06fa60fcd95f-kube-api-access-9nmnd\") pod \"217bd28e-15f2-42b3-8946-06fa60fcd95f\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") "
Nov 03 20:32:03.283649 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:03.283537    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-openshift-ca\") pod \"217bd28e-15f2-42b3-8946-06fa60fcd95f\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") "
Nov 03 20:32:03.283649 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:03.283601    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-xks-ca\") pod \"217bd28e-15f2-42b3-8946-06fa60fcd95f\" (UID: \"217bd28e-15f2-42b3-8946-06fa60fcd95f\") "
Nov 03 20:32:03.283903 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:03.283870    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-openshift-ca" pod "217bd28e-15f2-42b3-8946-06fa60fcd95f" (UID: "217bd28e-15f2-42b3-8946-06fa60fcd95f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:03.285794 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:03.285762    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-xks-ca" pod "217bd28e-15f2-42b3-8946-06fa60fcd95f" (UID: "217bd28e-15f2-42b3-8946-06fa60fcd95f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:03.285902 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:03.285799    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217bd28e-15f2-42b3-8946-06fa60fcd95f-kube-api-access-9nmnd" pod "217bd28e-15f2-42b3-8946-06fa60fcd95f" (UID: "217bd28e-15f2-42b3-8946-06fa60fcd95f"). InnerVolumeSpecName "kube-api-access-9nmnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:03.384783 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:03.384751    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:03.384783 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:03.384782    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/217bd28e-15f2-42b3-8946-06fa60fcd95f-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:03.384987 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:03.384791    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nmnd\" (UniqueName: \"kubernetes.io/projected/217bd28e-15f2-42b3-8946-06fa60fcd95f-kube-api-access-9nmnd\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:04.084708 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:04.084680    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"
Nov 03 20:32:04.084891 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:04.084698    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2" event={"ID":"217bd28e-15f2-42b3-8946-06fa60fcd95f","Type":"ContainerDied","Data":"79480e07d211f460438d504ed07125f5ca538c22c661c0475fcf5ea2eefe314f"}
Nov 03 20:32:04.084891 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:04.084751    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79480e07d211f460438d504ed07125f5ca538c22c661c0475fcf5ea2eefe314f"
Nov 03 20:32:23.444621 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:23.444564    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rrs66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:32:23.445311 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:23.445276    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:32:23.446105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:23.445965    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sk7xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:32:23.446338 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:23.446279    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:32:28.181053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.181018    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"]
Nov 03 20:32:28.181468 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.181428    5600 cpu_manager.go:414] "containerMap: removing container" podUID="217bd28e-15f2-42b3-8946-06fa60fcd95f" containerName="revoke-keys"
Nov 03 20:32:28.181532 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.181469    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="217bd28e-15f2-42b3-8946-06fa60fcd95f" containerName="revoke-keys"
Nov 03 20:32:28.181575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.181558    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="217bd28e-15f2-42b3-8946-06fa60fcd95f" containerName="revoke-keys"
Nov 03 20:32:28.184833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.184817    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:28.190544 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.190516    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:32:28.190686 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.190562    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:32:28.192235 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.192211    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"]
Nov 03 20:32:28.249360 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.249327    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jjr9n\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:28.249596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.249370    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jjr9n\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:28.249596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.249407    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2fwr\" (UniqueName: \"kubernetes.io/projected/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-kube-api-access-w2fwr\") pod \"maas-api-revoke-sub-e2fc9c5b-jjr9n\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:28.350893 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.350864    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jjr9n\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:28.351067 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.350904    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jjr9n\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:28.351067 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.350935    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2fwr\" (UniqueName: \"kubernetes.io/projected/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-kube-api-access-w2fwr\") pod \"maas-api-revoke-sub-e2fc9c5b-jjr9n\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:28.351574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.351540    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jjr9n\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:28.353334 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.353316    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jjr9n\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:28.360897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.360872    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2fwr\" (UniqueName: \"kubernetes.io/projected/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-kube-api-access-w2fwr\") pod \"maas-api-revoke-sub-e2fc9c5b-jjr9n\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:28.502063 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.501985    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:28.783034 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:28.783004    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"]
Nov 03 20:32:28.785326 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:32:28.785291    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a309a8_8cef_4bbe_ade9_2cfba1c8cd20.slice/crio-8c1f2d10607bba1b487346b30d2bc55311c941ae17077733ebef10a3911ebf68 WatchSource:0}: Error finding container 8c1f2d10607bba1b487346b30d2bc55311c941ae17077733ebef10a3911ebf68: Status 404 returned error can't find the container with id 8c1f2d10607bba1b487346b30d2bc55311c941ae17077733ebef10a3911ebf68
Nov 03 20:32:28.800385 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:32:28.800351    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:32:29.251085 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:29.251046    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n" event={"ID":"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20","Type":"ContainerStarted","Data":"f49be9b53883ee2ccb18a0f3549f449d48173aca7866975c62dc3b71f10cbb21"}
Nov 03 20:32:29.251530 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:29.251098    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n" event={"ID":"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20","Type":"ContainerStarted","Data":"8c1f2d10607bba1b487346b30d2bc55311c941ae17077733ebef10a3911ebf68"}
Nov 03 20:32:29.273688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:29.273642    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n" podStartSLOduration=1.273629434 podStartE2EDuration="1.273629434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:29.270550203 +0000 UTC m=+3131.783371985" watchObservedRunningTime="2026-11-03 20:32:29.273629434 +0000 UTC m=+3131.786451275"
Nov 03 20:32:30.032667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:30.032626    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"]
Nov 03 20:32:30.037693 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:30.037664    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9857b867-wdcz5"]
Nov 03 20:32:30.258157 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:30.258114    5600 generic.go:356] "Generic (PLEG): container finished" podID="c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20" containerID="f49be9b53883ee2ccb18a0f3549f449d48173aca7866975c62dc3b71f10cbb21" exitCode=0
Nov 03 20:32:30.258601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:30.258196    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n" event={"ID":"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20","Type":"ContainerDied","Data":"f49be9b53883ee2ccb18a0f3549f449d48173aca7866975c62dc3b71f10cbb21"}
Nov 03 20:32:31.398184 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:31.398149    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:31.479788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:31.479750    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-kube-api-access-w2fwr\" (UniqueName: \"kubernetes.io/projected/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-kube-api-access-w2fwr\") pod \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") "
Nov 03 20:32:31.479992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:31.479798    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-xks-ca\") pod \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") "
Nov 03 20:32:31.479992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:31.479836    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-openshift-ca\") pod \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\" (UID: \"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20\") "
Nov 03 20:32:31.480277 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:31.480253    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-openshift-ca" pod "c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20" (UID: "c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:31.482115 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:31.482093    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-xks-ca" pod "c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20" (UID: "c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:31.482192 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:31.482157    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-kube-api-access-w2fwr" pod "c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20" (UID: "c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20"). InnerVolumeSpecName "kube-api-access-w2fwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:31.581413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:31.581319    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2fwr\" (UniqueName: \"kubernetes.io/projected/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-kube-api-access-w2fwr\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:31.581413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:31.581350    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:31.581413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:31.581363    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:31.614965 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:31.614930    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6" path="/var/lib/kubelet/pods/5e5ded5b-4229-4c8a-bd2f-b6642dc7eac6/volumes"
Nov 03 20:32:32.270851 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:32.270814    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n" event={"ID":"c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20","Type":"ContainerDied","Data":"8c1f2d10607bba1b487346b30d2bc55311c941ae17077733ebef10a3911ebf68"}
Nov 03 20:32:32.270851 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:32.270834    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"
Nov 03 20:32:32.271134 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:32:32.270861    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c1f2d10607bba1b487346b30d2bc55311c941ae17077733ebef10a3911ebf68"
Nov 03 20:32:58.795067 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:32:58.794976    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:33:22.094942 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:33:22.094906    5600 scope.go:122] "RemoveContainer" containerID="9359290a817b482dca699351688522824e1d90e39427a03b76672f19bb4042a4"
Nov 03 20:33:22.104108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:33:22.104085    5600 scope.go:122] "RemoveContainer" containerID="46c05838ca215d7ebbbb02fccbc3ef661bdb71fb22500a22996f616dca1acb29"
Nov 03 20:33:28.794778 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:33:28.794739    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:33:58.797669 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:33:58.797622    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:34:22.129730 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:34:22.129689    5600 scope.go:122] "RemoveContainer" containerID="c44b80c2a8d75ebf45132fdb5e836d9975e22bae7cc601e16ef91864acd28c32"
Nov 03 20:34:28.795351 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:34:28.795264    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:34:58.794698 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:34:58.794657    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:35:09.019524 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:35:09.019488    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"]
Nov 03 20:35:09.022055 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:35:09.022036    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2n858"]
Nov 03 20:35:09.615033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:35:09.615001    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b2f190-16d9-49c1-bc2e-5718c31450aa" path="/var/lib/kubelet/pods/96b2f190-16d9-49c1-bc2e-5718c31450aa/volumes"
Nov 03 20:35:19.020481 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:35:19.020445    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"]
Nov 03 20:35:19.024970 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:35:19.024944    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-p52xw"]
Nov 03 20:35:19.615371 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:35:19.615335    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9a642d-b921-4d96-a54c-0ce3164651b6" path="/var/lib/kubelet/pods/cd9a642d-b921-4d96-a54c-0ce3164651b6/volumes"
Nov 03 20:35:28.794888 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:35:28.794849    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:35:58.794973 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:35:58.794890    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:36:22.165296 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:36:22.165258    5600 scope.go:122] "RemoveContainer" containerID="f2780ce30783555ffbb5096cee9564ad37ee8ea270277fc5b86e84ea35991cc2"
Nov 03 20:36:22.178006 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:36:22.177982    5600 scope.go:122] "RemoveContainer" containerID="97c561edf0c84949a6dd424500e2c96a8577b4ae158231f177635fd834191753"
Nov 03 20:36:28.794796 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:36:28.794755    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:36:35.021108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:36:35.021068    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"]
Nov 03 20:36:35.024482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:36:35.024456    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rl5kv"]
Nov 03 20:36:35.615159 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:36:35.615122    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6e2559-1e92-4e53-97c3-7b1273464d37" path="/var/lib/kubelet/pods/4e6e2559-1e92-4e53-97c3-7b1273464d37/volumes"
Nov 03 20:36:58.794044 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:36:58.794000    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:37:04.019392 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:37:04.019354    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"]
Nov 03 20:37:04.021847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:37:04.021821    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rfhn2"]
Nov 03 20:37:05.614714 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:37:05.614680    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="217bd28e-15f2-42b3-8946-06fa60fcd95f" path="/var/lib/kubelet/pods/217bd28e-15f2-42b3-8946-06fa60fcd95f/volumes"
Nov 03 20:37:28.793540 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:37:28.793459    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:37:32.022312 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:37:32.022272    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"]
Nov 03 20:37:32.024608 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:37:32.024583    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jjr9n"]
Nov 03 20:37:33.615280 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:37:33.615246    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20" path="/var/lib/kubelet/pods/c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20/volumes"
Nov 03 20:37:58.794157 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:37:58.794111    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:38:03.405334 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:03.405286    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"]
Nov 03 20:38:03.405981 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:03.405559    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy" containerID="cri-o://d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6" gracePeriod=30
Nov 03 20:38:03.408345 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:03.407921    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"]
Nov 03 20:38:03.408345 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:03.408176    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy" containerID="cri-o://4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4" gracePeriod=30
Nov 03 20:38:03.410094 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:03.410070    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"]
Nov 03 20:38:03.410282 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:03.410258    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy" containerID="cri-o://d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed" gracePeriod=30
Nov 03 20:38:08.426521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.425932    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rrs66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:38:08.426521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.426101    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:38:08.438124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.437899    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sk7xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:38:08.438124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.438062    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:38:08.540046 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:38:08.540016    5600 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c3aae1_9313_4ddd_a223_c6feefd0eaf6.slice/crio-conmon-4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c3aae1_9313_4ddd_a223_c6feefd0eaf6.slice/crio-4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:38:08.685731 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.685670    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 20:38:08.722273 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.722244    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 20:38:08.727671 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.727208    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:38:08.753426 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753398    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-token\" (UniqueName: \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-token\") pod \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") "
Nov 03 20:38:08.753570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753456    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-data\") pod \"cead9ddc-269e-4641-80a7-30b294529b02\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") "
Nov 03 20:38:08.753570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753539    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-envoy\") pod \"cead9ddc-269e-4641-80a7-30b294529b02\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") "
Nov 03 20:38:08.753693 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753666    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istiod-ca-cert\") pod \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") "
Nov 03 20:38:08.753757 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753670    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-data" pod "cead9ddc-269e-4641-80a7-30b294529b02" (UID: "cead9ddc-269e-4641-80a7-30b294529b02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.753818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753754    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cead9ddc-269e-4641-80a7-30b294529b02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cead9ddc-269e-4641-80a7-30b294529b02-istio-podinfo\") pod \"cead9ddc-269e-4641-80a7-30b294529b02\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") "
Nov 03 20:38:08.753818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753803    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-envoy\") pod \"aea46236-0353-41e5-9382-9408c9398b00\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") "
Nov 03 20:38:08.753950 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753838    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-credential-socket\") pod \"cead9ddc-269e-4641-80a7-30b294529b02\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") "
Nov 03 20:38:08.753950 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753882    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-istio-token\" (UniqueName: \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-istio-token\") pod \"cead9ddc-269e-4641-80a7-30b294529b02\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") "
Nov 03 20:38:08.753950 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753913    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-socket\") pod \"cead9ddc-269e-4641-80a7-30b294529b02\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") "
Nov 03 20:38:08.753950 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753941    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-istio-token\" (UniqueName: \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-istio-token\") pod \"aea46236-0353-41e5-9382-9408c9398b00\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") "
Nov 03 20:38:08.754254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753968    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-data\") pod \"aea46236-0353-41e5-9382-9408c9398b00\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") "
Nov 03 20:38:08.754254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754001    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-certs\") pod \"aea46236-0353-41e5-9382-9408c9398b00\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") "
Nov 03 20:38:08.754254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754040    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-kube-api-access-x247v\" (UniqueName: \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-kube-api-access-x247v\") pod \"aea46236-0353-41e5-9382-9408c9398b00\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") "
Nov 03 20:38:08.754254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754072    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-certs\") pod \"cead9ddc-269e-4641-80a7-30b294529b02\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") "
Nov 03 20:38:08.754254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754102    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-credential-socket\") pod \"aea46236-0353-41e5-9382-9408c9398b00\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") "
Nov 03 20:38:08.754254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754131    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-envoy\") pod \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") "
Nov 03 20:38:08.754254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754178    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-kube-api-access-c874g\" (UniqueName: \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-kube-api-access-c874g\") pod \"cead9ddc-269e-4641-80a7-30b294529b02\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") "
Nov 03 20:38:08.754254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754206    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cead9ddc-269e-4641-80a7-30b294529b02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cead9ddc-269e-4641-80a7-30b294529b02-istiod-ca-cert\") pod \"cead9ddc-269e-4641-80a7-30b294529b02\" (UID: \"cead9ddc-269e-4641-80a7-30b294529b02\") "
Nov 03 20:38:08.754254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754250    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-kube-api-access-b2bcx\" (UniqueName: \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-kube-api-access-b2bcx\") pod \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") "
Nov 03 20:38:08.754736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754306    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-data\") pod \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") "
Nov 03 20:38:08.754736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754340    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-socket\") pod \"aea46236-0353-41e5-9382-9408c9398b00\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") "
Nov 03 20:38:08.754736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754371    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea46236-0353-41e5-9382-9408c9398b00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea46236-0353-41e5-9382-9408c9398b00-istiod-ca-cert\") pod \"aea46236-0353-41e5-9382-9408c9398b00\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") "
Nov 03 20:38:08.754736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754412    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aea46236-0353-41e5-9382-9408c9398b00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea46236-0353-41e5-9382-9408c9398b00-istio-podinfo\") pod \"aea46236-0353-41e5-9382-9408c9398b00\" (UID: \"aea46236-0353-41e5-9382-9408c9398b00\") "
Nov 03 20:38:08.754736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754456    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-socket\") pod \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") "
Nov 03 20:38:08.754990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754815    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.755393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.753999    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istiod-ca-cert" pod "61c3aae1-9313-4ddd-a223-c6feefd0eaf6" (UID: "61c3aae1-9313-4ddd-a223-c6feefd0eaf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:08.755393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.754729    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-credential-socket" pod "aea46236-0353-41e5-9382-9408c9398b00" (UID: "aea46236-0353-41e5-9382-9408c9398b00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.755393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.755024    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-socket" pod "61c3aae1-9313-4ddd-a223-c6feefd0eaf6" (UID: "61c3aae1-9313-4ddd-a223-c6feefd0eaf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.755393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.755043    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-socket" pod "cead9ddc-269e-4641-80a7-30b294529b02" (UID: "cead9ddc-269e-4641-80a7-30b294529b02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.755703 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.755534    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-socket" pod "aea46236-0353-41e5-9382-9408c9398b00" (UID: "aea46236-0353-41e5-9382-9408c9398b00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.756074 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.755799    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-data" pod "aea46236-0353-41e5-9382-9408c9398b00" (UID: "aea46236-0353-41e5-9382-9408c9398b00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.756074 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.755809    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea46236-0353-41e5-9382-9408c9398b00-istiod-ca-cert" pod "aea46236-0353-41e5-9382-9408c9398b00" (UID: "aea46236-0353-41e5-9382-9408c9398b00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:08.756074 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.755948    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-certs" pod "cead9ddc-269e-4641-80a7-30b294529b02" (UID: "cead9ddc-269e-4641-80a7-30b294529b02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.756074 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.756012    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-certs" pod "aea46236-0353-41e5-9382-9408c9398b00" (UID: "aea46236-0353-41e5-9382-9408c9398b00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.756357 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.756098    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-data" pod "61c3aae1-9313-4ddd-a223-c6feefd0eaf6" (UID: "61c3aae1-9313-4ddd-a223-c6feefd0eaf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.756357 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.756156    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-credential-socket" pod "cead9ddc-269e-4641-80a7-30b294529b02" (UID: "cead9ddc-269e-4641-80a7-30b294529b02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.756357 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.756349    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cead9ddc-269e-4641-80a7-30b294529b02-istiod-ca-cert" pod "cead9ddc-269e-4641-80a7-30b294529b02" (UID: "cead9ddc-269e-4641-80a7-30b294529b02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:08.757261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.756925    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-istio-token" pod "cead9ddc-269e-4641-80a7-30b294529b02" (UID: "cead9ddc-269e-4641-80a7-30b294529b02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.757261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.757069    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-token" pod "61c3aae1-9313-4ddd-a223-c6feefd0eaf6" (UID: "61c3aae1-9313-4ddd-a223-c6feefd0eaf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.757261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.757178    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-envoy" pod "cead9ddc-269e-4641-80a7-30b294529b02" (UID: "cead9ddc-269e-4641-80a7-30b294529b02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.757968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.757886    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cead9ddc-269e-4641-80a7-30b294529b02-istio-podinfo" pod "cead9ddc-269e-4641-80a7-30b294529b02" (UID: "cead9ddc-269e-4641-80a7-30b294529b02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:08.759873 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.759834    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-kube-api-access-c874g" pod "cead9ddc-269e-4641-80a7-30b294529b02" (UID: "cead9ddc-269e-4641-80a7-30b294529b02"). InnerVolumeSpecName "kube-api-access-c874g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.760194 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.760147    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-kube-api-access-b2bcx" pod "61c3aae1-9313-4ddd-a223-c6feefd0eaf6" (UID: "61c3aae1-9313-4ddd-a223-c6feefd0eaf6"). InnerVolumeSpecName "kube-api-access-b2bcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.760356 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.760321    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-kube-api-access-x247v" pod "aea46236-0353-41e5-9382-9408c9398b00" (UID: "aea46236-0353-41e5-9382-9408c9398b00"). InnerVolumeSpecName "kube-api-access-x247v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.760838 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.760812    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aea46236-0353-41e5-9382-9408c9398b00-istio-podinfo" pod "aea46236-0353-41e5-9382-9408c9398b00" (UID: "aea46236-0353-41e5-9382-9408c9398b00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:08.760992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.760834    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-istio-token" pod "aea46236-0353-41e5-9382-9408c9398b00" (UID: "aea46236-0353-41e5-9382-9408c9398b00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.761107 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.761065    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-envoy" pod "aea46236-0353-41e5-9382-9408c9398b00" (UID: "aea46236-0353-41e5-9382-9408c9398b00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.761176 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.761113    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-envoy" pod "61c3aae1-9313-4ddd-a223-c6feefd0eaf6" (UID: "61c3aae1-9313-4ddd-a223-c6feefd0eaf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.855834 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.855801    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-certs\") pod \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") "
Nov 03 20:38:08.856038 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.855849    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-credential-socket\") pod \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") "
Nov 03 20:38:08.856038 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.855872    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-podinfo\") pod \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\" (UID: \"61c3aae1-9313-4ddd-a223-c6feefd0eaf6\") "
Nov 03 20:38:08.856172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856086    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856096    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856095    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-credential-socket" pod "61c3aae1-9313-4ddd-a223-c6feefd0eaf6" (UID: "61c3aae1-9313-4ddd-a223-c6feefd0eaf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.856172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856106    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856148    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856159    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856172 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856170    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x247v\" (UniqueName: \"kubernetes.io/projected/aea46236-0353-41e5-9382-9408c9398b00-kube-api-access-x247v\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856180    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856189    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856121    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-certs" pod "61c3aae1-9313-4ddd-a223-c6feefd0eaf6" (UID: "61c3aae1-9313-4ddd-a223-c6feefd0eaf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856197    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856235    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c874g\" (UniqueName: \"kubernetes.io/projected/cead9ddc-269e-4641-80a7-30b294529b02-kube-api-access-c874g\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856254    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cead9ddc-269e-4641-80a7-30b294529b02-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856269    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2bcx\" (UniqueName: \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-kube-api-access-b2bcx\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856283    5600 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-data\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856299    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856313    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea46236-0353-41e5-9382-9408c9398b00-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856327    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea46236-0353-41e5-9382-9408c9398b00-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856341    5600 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856357    5600 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-token\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856370    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856382    5600 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istiod-ca-cert\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856395    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cead9ddc-269e-4641-80a7-30b294529b02-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856409    5600 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea46236-0353-41e5-9382-9408c9398b00-istio-envoy\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.856558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.856422    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cead9ddc-269e-4641-80a7-30b294529b02-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.858054 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.858035    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-podinfo" pod "61c3aae1-9313-4ddd-a223-c6feefd0eaf6" (UID: "61c3aae1-9313-4ddd-a223-c6feefd0eaf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:08.957617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.957539    5600 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-workload-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.957617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.957570    5600 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-credential-socket\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.957617 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:08.957580    5600 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c3aae1-9313-4ddd-a223-c6feefd0eaf6-istio-podinfo\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:09.074454 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.074395    5600 generic.go:356] "Generic (PLEG): container finished" podID="aea46236-0353-41e5-9382-9408c9398b00" containerID="d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6" exitCode=0
Nov 03 20:38:09.074662 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.074467    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"
Nov 03 20:38:09.074662 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.074493    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" event={"ID":"aea46236-0353-41e5-9382-9408c9398b00","Type":"ContainerDied","Data":"d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6"}
Nov 03 20:38:09.074662 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.074552    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk" event={"ID":"aea46236-0353-41e5-9382-9408c9398b00","Type":"ContainerDied","Data":"c417d9b73e9db3587deb35cd5b84e0fca3b0c2e2c32a2b8d76d791afb6963748"}
Nov 03 20:38:09.074662 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.074582    5600 scope.go:122] "RemoveContainer" containerID="d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6"
Nov 03 20:38:09.075982 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.075949    5600 generic.go:356] "Generic (PLEG): container finished" podID="cead9ddc-269e-4641-80a7-30b294529b02" containerID="d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed" exitCode=0
Nov 03 20:38:09.075982 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.075976    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" event={"ID":"cead9ddc-269e-4641-80a7-30b294529b02","Type":"ContainerDied","Data":"d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed"}
Nov 03 20:38:09.076175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.075997    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"
Nov 03 20:38:09.076175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.076016    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66" event={"ID":"cead9ddc-269e-4641-80a7-30b294529b02","Type":"ContainerDied","Data":"7a95b57914162d4969c93cb96abb61a7b5c86f65ea3e29701a5a0872dc5ddb6c"}
Nov 03 20:38:09.077638 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.077614    5600 generic.go:356] "Generic (PLEG): container finished" podID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerID="4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4" exitCode=0
Nov 03 20:38:09.077763 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.077645    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" event={"ID":"61c3aae1-9313-4ddd-a223-c6feefd0eaf6","Type":"ContainerDied","Data":"4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4"}
Nov 03 20:38:09.077763 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.077664    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"
Nov 03 20:38:09.077763 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.077696    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd" event={"ID":"61c3aae1-9313-4ddd-a223-c6feefd0eaf6","Type":"ContainerDied","Data":"c4a130556fc0e3e61fdf4603a8fa0c7443a556a75979f0424a15a30440a07f37"}
Nov 03 20:38:09.084956 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.084935    5600 scope.go:122] "RemoveContainer" containerID="d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6"
Nov 03 20:38:09.085351 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:38:09.085329    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6\": container with ID starting with d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6 not found: ID does not exist" containerID="d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6"
Nov 03 20:38:09.085534 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.085356    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6"} err="failed to get container status \"d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6\": rpc error: code = NotFound desc = could not find container \"d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6\": container with ID starting with d648d0a0a3f5e593272e3bedcfe8cd872c242ca02a08374717cfb178dfdcc0c6 not found: ID does not exist"
Nov 03 20:38:09.085534 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.085372    5600 scope.go:122] "RemoveContainer" containerID="d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed"
Nov 03 20:38:09.095052 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.095034    5600 scope.go:122] "RemoveContainer" containerID="d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed"
Nov 03 20:38:09.095295 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:38:09.095274    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed\": container with ID starting with d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed not found: ID does not exist" containerID="d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed"
Nov 03 20:38:09.095343 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.095304    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed"} err="failed to get container status \"d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed\": rpc error: code = NotFound desc = could not find container \"d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed\": container with ID starting with d0c442d39c120a3044b0e751b5a1d20e6157974632cff1706b03708acb7946ed not found: ID does not exist"
Nov 03 20:38:09.095343 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.095319    5600 scope.go:122] "RemoveContainer" containerID="4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4"
Nov 03 20:38:09.103400 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.103314    5600 scope.go:122] "RemoveContainer" containerID="4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4"
Nov 03 20:38:09.103672 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:38:09.103652    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4\": container with ID starting with 4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4 not found: ID does not exist" containerID="4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4"
Nov 03 20:38:09.103763 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.103678    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4"} err="failed to get container status \"4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4\": rpc error: code = NotFound desc = could not find container \"4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4\": container with ID starting with 4dd800b97d7fa7e82ab8e1f4da6dc4d953ef427a6057019310b4af834900cac4 not found: ID does not exist"
Nov 03 20:38:09.104943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.104925    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"]
Nov 03 20:38:09.111864 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.111845    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rrs66"]
Nov 03 20:38:09.127564 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.127540    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"]
Nov 03 20:38:09.131121 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.131101    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sk7xk"]
Nov 03 20:38:09.142741 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.142704    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"]
Nov 03 20:38:09.146733 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.146709    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vbhfd"]
Nov 03 20:38:09.617121 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.617089    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" path="/var/lib/kubelet/pods/61c3aae1-9313-4ddd-a223-c6feefd0eaf6/volumes"
Nov 03 20:38:09.617532 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.617520    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea46236-0353-41e5-9382-9408c9398b00" path="/var/lib/kubelet/pods/aea46236-0353-41e5-9382-9408c9398b00/volumes"
Nov 03 20:38:09.617904 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:09.617891    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cead9ddc-269e-4641-80a7-30b294529b02" path="/var/lib/kubelet/pods/cead9ddc-269e-4641-80a7-30b294529b02/volumes"
Nov 03 20:38:18.452632 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.452595    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"]
Nov 03 20:38:18.453105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453027    5600 cpu_manager.go:414] "containerMap: removing container" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy"
Nov 03 20:38:18.453105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453042    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy"
Nov 03 20:38:18.453105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453057    5600 cpu_manager.go:414] "containerMap: removing container" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy"
Nov 03 20:38:18.453105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453062    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy"
Nov 03 20:38:18.453105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453069    5600 cpu_manager.go:414] "containerMap: removing container" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy"
Nov 03 20:38:18.453105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453074    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy"
Nov 03 20:38:18.453105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453087    5600 cpu_manager.go:414] "containerMap: removing container" podUID="c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20" containerName="revoke-keys"
Nov 03 20:38:18.453105 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453091    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20" containerName="revoke-keys"
Nov 03 20:38:18.453354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453149    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="cead9ddc-269e-4641-80a7-30b294529b02" containerName="istio-proxy"
Nov 03 20:38:18.453354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453158    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea46236-0353-41e5-9382-9408c9398b00" containerName="istio-proxy"
Nov 03 20:38:18.453354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453163    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c3aae1-9313-4ddd-a223-c6feefd0eaf6" containerName="istio-proxy"
Nov 03 20:38:18.453354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.453170    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a309a8-8cef-4bbe-ade9-2cfba1c8cd20" containerName="revoke-keys"
Nov 03 20:38:18.456214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.456198    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.465636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.465583    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 20:38:18.469295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.469260    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"]
Nov 03 20:38:18.541087 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.541053    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dkm76\""
Nov 03 20:38:18.546590 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.546564    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41ec2f8-05e2-456f-be60-a013db04b022-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.546752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.546616    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.546752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.546692    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.546752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.546733    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.546902 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.546799    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxx4\" (UniqueName: \"kubernetes.io/projected/e41ec2f8-05e2-456f-be60-a013db04b022-kube-api-access-kqxx4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.546902 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.546838    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.546902 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.546873    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41ec2f8-05e2-456f-be60-a013db04b022-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.547020 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.546918    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.547020 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.546945    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41ec2f8-05e2-456f-be60-a013db04b022-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.648113 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.648075    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41ec2f8-05e2-456f-be60-a013db04b022-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.648331 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.648138    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.648331 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.648314    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.648534 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.648511    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.648609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.648541    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxx4\" (UniqueName: \"kubernetes.io/projected/e41ec2f8-05e2-456f-be60-a013db04b022-kube-api-access-kqxx4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.648609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.648555    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.648609 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.648565    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.648754 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.648631    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41ec2f8-05e2-456f-be60-a013db04b022-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.648790 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.648756    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.648848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.648827    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41ec2f8-05e2-456f-be60-a013db04b022-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.649463 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.649409    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.649574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.649461    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.649574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.649515    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.649699 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.649664    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41ec2f8-05e2-456f-be60-a013db04b022-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.650855 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.650832    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41ec2f8-05e2-456f-be60-a013db04b022-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.652154 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.652137    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41ec2f8-05e2-456f-be60-a013db04b022-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.655826 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.655802    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxx4\" (UniqueName: \"kubernetes.io/projected/e41ec2f8-05e2-456f-be60-a013db04b022-kube-api-access-kqxx4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.657043 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.657022    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41ec2f8-05e2-456f-be60-a013db04b022-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4f6wl\" (UID: \"e41ec2f8-05e2-456f-be60-a013db04b022\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:18.774723 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:18.774649    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:19.070970 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:19.070788    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"]
Nov 03 20:38:19.073560 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:38:19.073532    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41ec2f8_05e2_456f_be60_a013db04b022.slice/crio-4eefb013439a9be3e535fb4957812a73f4e75d5e6090939b095c424ae5787224 WatchSource:0}: Error finding container 4eefb013439a9be3e535fb4957812a73f4e75d5e6090939b095c424ae5787224: Status 404 returned error can't find the container with id 4eefb013439a9be3e535fb4957812a73f4e75d5e6090939b095c424ae5787224
Nov 03 20:38:19.075783 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:19.075751    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:38:19.075890 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:19.075828    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:38:19.075890 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:19.075872    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:38:19.132667 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:19.132642    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl" event={"ID":"e41ec2f8-05e2-456f-be60-a013db04b022","Type":"ContainerStarted","Data":"4eefb013439a9be3e535fb4957812a73f4e75d5e6090939b095c424ae5787224"}
Nov 03 20:38:20.140340 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:20.140299    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl" event={"ID":"e41ec2f8-05e2-456f-be60-a013db04b022","Type":"ContainerStarted","Data":"3c59005d9b877bef2fd93e4c3fd2789b6dbe9e3d462e8846cb27bafa85544b45"}
Nov 03 20:38:20.164915 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:20.164859    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl" podStartSLOduration=2.164845494 podStartE2EDuration="2.164845494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:20.161385369 +0000 UTC m=+3482.674207161" watchObservedRunningTime="2026-11-03 20:38:20.164845494 +0000 UTC m=+3482.677667275"
Nov 03 20:38:20.775851 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:20.775812    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:20.777312 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:20.777287    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4f6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:38:20.777519 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:20.777333    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl" podUID="e41ec2f8-05e2-456f-be60-a013db04b022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 20:38:21.775972 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:21.775937    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4f6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:38:21.776357 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:21.775990    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl" podUID="e41ec2f8-05e2-456f-be60-a013db04b022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 20:38:22.218120 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:22.218073    5600 scope.go:122] "RemoveContainer" containerID="c719bb3e39493f5250aaa29a39d0dabad7419bd2aa87f873611e66c1b425d4b8"
Nov 03 20:38:22.227920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:22.227896    5600 scope.go:122] "RemoveContainer" containerID="bff9c8daf0befd3d0382155f360e124392a55b7540eac343a9fa9b644449aae4"
Nov 03 20:38:22.238842 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:22.238819    5600 scope.go:122] "RemoveContainer" containerID="3f0522a1c38a937e600c0f0e4979bedaaa80445bfc05b436eac39e2164c42311"
Nov 03 20:38:22.775092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:22.775057    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4f6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:38:22.775276 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:22.775109    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl" podUID="e41ec2f8-05e2-456f-be60-a013db04b022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 20:38:23.775972 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:23.775930    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4f6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:38:23.776494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:23.775981    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl" podUID="e41ec2f8-05e2-456f-be60-a013db04b022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 20:38:24.776033 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:24.775998    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4f6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:38:24.776560 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:24.776068    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl" podUID="e41ec2f8-05e2-456f-be60-a013db04b022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 20:38:25.776025 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:25.775984    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4f6wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:38:25.776212 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:25.776055    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl" podUID="e41ec2f8-05e2-456f-be60-a013db04b022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 20:38:27.116293 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:27.116257    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:27.116842 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:27.116806    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:27.117592 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:27.117575    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4f6wl"
Nov 03 20:38:28.794760 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:38:28.794718    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:38:34.573747 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.573710    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"]
Nov 03 20:38:34.580101 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.580075    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.585486 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.585459    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"]
Nov 03 20:38:34.587408 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.587384    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4442a4\"/\"openshift-service-ca.crt\""
Nov 03 20:38:34.587575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.587387    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4442a4\"/\"kube-root-ca.crt\""
Nov 03 20:38:34.602969 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.602941    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4442a4\"/\"simulator-single-main-4442a4-kserve-self-signed-certs\""
Nov 03 20:38:34.603187 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.602970    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4442a4\"/\"default-dockercfg-ph7nk\""
Nov 03 20:38:34.730248 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.730210    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-dshm\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.730427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.730260    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-model-cache\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.730427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.730339    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tmp-dir\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.730427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.730416    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gl9\" (UniqueName: \"kubernetes.io/projected/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-kube-api-access-j4gl9\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.730582 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.730473    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tls-certs\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.730582 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.730512    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-home\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.831829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.831735    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tls-certs\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.831829 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.831814    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-home\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.832081 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.831853    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-dshm\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.832081 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.831898    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-model-cache\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.832081 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.831933    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tmp-dir\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.832081 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.831973    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gl9\" (UniqueName: \"kubernetes.io/projected/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-kube-api-access-j4gl9\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.832336 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.832313    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tmp-dir\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.832421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.832370    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-model-cache\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.832421 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.832401    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-home\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.834345 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.834313    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-dshm\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.834741 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.834721    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tls-certs\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.841220 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.841194    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gl9\" (UniqueName: \"kubernetes.io/projected/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-kube-api-access-j4gl9\") pod \"simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") " pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:34.899091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:34.899055    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:35.175902 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:35.175879    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"]
Nov 03 20:38:35.178539 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:38:35.178510    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1fd8f08_a3ab_4bde_ac57_f77cf2b5f0d0.slice/crio-24bb70489ae9775202b0133b207a7ae08d9ca18c0181563fba0ff8dd3f36b73d WatchSource:0}: Error finding container 24bb70489ae9775202b0133b207a7ae08d9ca18c0181563fba0ff8dd3f36b73d: Status 404 returned error can't find the container with id 24bb70489ae9775202b0133b207a7ae08d9ca18c0181563fba0ff8dd3f36b73d
Nov 03 20:38:35.223140 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:35.223104    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf" event={"ID":"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0","Type":"ContainerStarted","Data":"24bb70489ae9775202b0133b207a7ae08d9ca18c0181563fba0ff8dd3f36b73d"}
Nov 03 20:38:36.230891 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:36.230851    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf" event={"ID":"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0","Type":"ContainerStarted","Data":"47c89a80ca400a6d66daa06b639be8bcaa74959f08f6603657fa5b29fecd3c4a"}
Nov 03 20:38:36.250018 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:36.249959    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf" podStartSLOduration=2.249940731 podStartE2EDuration="2.249940731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:36.248317856 +0000 UTC m=+3498.761139638" watchObservedRunningTime="2026-11-03 20:38:36.249940731 +0000 UTC m=+3498.762762514"
Nov 03 20:38:36.899543 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:36.899503    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:44.899970 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:44.899930    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:44.912343 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:44.912315    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:45.288661 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:38:45.288586    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:38:58.794222 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:38:58.794185    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:39:04.515126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.515085    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"]
Nov 03 20:39:04.518659 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.518639    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:04.523606 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.523545    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:39:04.524547 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.524523    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:39:04.527075 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.527052    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"]
Nov 03 20:39:04.593781 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.593745    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2d6539-bhw79\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:04.593781 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.593793    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2d6539-bhw79\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:04.594059 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.594001    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qhtk\" (UniqueName: \"kubernetes.io/projected/cbc53c6e-96d6-4907-b241-a6d89a05abf7-kube-api-access-5qhtk\") pod \"maas-api-revoke-sub-0f2d6539-bhw79\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:04.695166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.695131    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qhtk\" (UniqueName: \"kubernetes.io/projected/cbc53c6e-96d6-4907-b241-a6d89a05abf7-kube-api-access-5qhtk\") pod \"maas-api-revoke-sub-0f2d6539-bhw79\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:04.695166 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.695166    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2d6539-bhw79\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:04.695456 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.695284    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2d6539-bhw79\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:04.695918 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.695893    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2d6539-bhw79\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:04.697792 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.697774    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2d6539-bhw79\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:04.703076 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.703058    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qhtk\" (UniqueName: \"kubernetes.io/projected/cbc53c6e-96d6-4907-b241-a6d89a05abf7-kube-api-access-5qhtk\") pod \"maas-api-revoke-sub-0f2d6539-bhw79\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:04.833647 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:04.833558    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:05.106253 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:05.106228    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"]
Nov 03 20:39:05.108761 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:39:05.108733    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc53c6e_96d6_4907_b241_a6d89a05abf7.slice/crio-a11f17e2a1315c834d95b3ff55795bc28b7329eeec06d90362fd6556ddad1c7d WatchSource:0}: Error finding container a11f17e2a1315c834d95b3ff55795bc28b7329eeec06d90362fd6556ddad1c7d: Status 404 returned error can't find the container with id a11f17e2a1315c834d95b3ff55795bc28b7329eeec06d90362fd6556ddad1c7d
Nov 03 20:39:05.376943 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:05.376859    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79" event={"ID":"cbc53c6e-96d6-4907-b241-a6d89a05abf7","Type":"ContainerStarted","Data":"a11f17e2a1315c834d95b3ff55795bc28b7329eeec06d90362fd6556ddad1c7d"}
Nov 03 20:39:06.383282 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:06.383199    5600 generic.go:356] "Generic (PLEG): container finished" podID="cbc53c6e-96d6-4907-b241-a6d89a05abf7" containerID="abadb878f53f76f64bbe920059e9d417c003f9558d09703def8683d1213fce55" exitCode=0
Nov 03 20:39:06.383657 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:06.383283    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79" event={"ID":"cbc53c6e-96d6-4907-b241-a6d89a05abf7","Type":"ContainerDied","Data":"abadb878f53f76f64bbe920059e9d417c003f9558d09703def8683d1213fce55"}
Nov 03 20:39:07.505800 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:07.505779    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:07.623031 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:07.623001    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-xks-ca\") pod \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") "
Nov 03 20:39:07.623203 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:07.623063    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-openshift-ca\") pod \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") "
Nov 03 20:39:07.623203 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:07.623140    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc53c6e-96d6-4907-b241-a6d89a05abf7-kube-api-access-5qhtk\" (UniqueName: \"kubernetes.io/projected/cbc53c6e-96d6-4907-b241-a6d89a05abf7-kube-api-access-5qhtk\") pod \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\" (UID: \"cbc53c6e-96d6-4907-b241-a6d89a05abf7\") "
Nov 03 20:39:07.623393 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:07.623370    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-openshift-ca" pod "cbc53c6e-96d6-4907-b241-a6d89a05abf7" (UID: "cbc53c6e-96d6-4907-b241-a6d89a05abf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:07.625312 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:07.625285    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-xks-ca" pod "cbc53c6e-96d6-4907-b241-a6d89a05abf7" (UID: "cbc53c6e-96d6-4907-b241-a6d89a05abf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:07.625525 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:07.625507    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc53c6e-96d6-4907-b241-a6d89a05abf7-kube-api-access-5qhtk" pod "cbc53c6e-96d6-4907-b241-a6d89a05abf7" (UID: "cbc53c6e-96d6-4907-b241-a6d89a05abf7"). InnerVolumeSpecName "kube-api-access-5qhtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:07.724413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:07.724312    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:07.724413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:07.724352    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qhtk\" (UniqueName: \"kubernetes.io/projected/cbc53c6e-96d6-4907-b241-a6d89a05abf7-kube-api-access-5qhtk\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:07.724413 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:07.724366    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc53c6e-96d6-4907-b241-a6d89a05abf7-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:08.394169 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:08.394127    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"
Nov 03 20:39:08.394169 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:08.394158    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79" event={"ID":"cbc53c6e-96d6-4907-b241-a6d89a05abf7","Type":"ContainerDied","Data":"a11f17e2a1315c834d95b3ff55795bc28b7329eeec06d90362fd6556ddad1c7d"}
Nov 03 20:39:08.394427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:08.394206    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a11f17e2a1315c834d95b3ff55795bc28b7329eeec06d90362fd6556ddad1c7d"
Nov 03 20:39:15.566929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.566894    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"]
Nov 03 20:39:15.567508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.567491    5600 cpu_manager.go:414] "containerMap: removing container" podUID="cbc53c6e-96d6-4907-b241-a6d89a05abf7" containerName="revoke-keys"
Nov 03 20:39:15.567571 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.567510    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc53c6e-96d6-4907-b241-a6d89a05abf7" containerName="revoke-keys"
Nov 03 20:39:15.567624 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.567614    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc53c6e-96d6-4907-b241-a6d89a05abf7" containerName="revoke-keys"
Nov 03 20:39:15.571316 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.571298    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:15.577074 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.577048    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"]
Nov 03 20:39:15.585515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.585281    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:39:15.585515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.585282    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:39:15.698753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.698712    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5xh9\" (UniqueName: \"kubernetes.io/projected/4fa26111-5767-466d-94f7-cb2b5e79690a-kube-api-access-h5xh9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:15.698937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.698776    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:15.698937 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.698868    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:15.799665 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.799628    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:15.799847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.799745    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5xh9\" (UniqueName: \"kubernetes.io/projected/4fa26111-5767-466d-94f7-cb2b5e79690a-kube-api-access-h5xh9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:15.799847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.799776    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:15.800556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.800485    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:15.802293 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.802264    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:15.807987 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.807957    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5xh9\" (UniqueName: \"kubernetes.io/projected/4fa26111-5767-466d-94f7-cb2b5e79690a-kube-api-access-h5xh9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:15.887899 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:15.887863    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:16.163897 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:16.163867    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"]
Nov 03 20:39:16.165952 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:39:16.165920    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa26111_5767_466d_94f7_cb2b5e79690a.slice/crio-1f61bf38423e73ade8d793b7ac02713ce053c52e522cb4389065c82ea89e1c56 WatchSource:0}: Error finding container 1f61bf38423e73ade8d793b7ac02713ce053c52e522cb4389065c82ea89e1c56: Status 404 returned error can't find the container with id 1f61bf38423e73ade8d793b7ac02713ce053c52e522cb4389065c82ea89e1c56
Nov 03 20:39:16.433867 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:16.433785    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5" event={"ID":"4fa26111-5767-466d-94f7-cb2b5e79690a","Type":"ContainerStarted","Data":"1f61bf38423e73ade8d793b7ac02713ce053c52e522cb4389065c82ea89e1c56"}
Nov 03 20:39:17.441533 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:17.441494    5600 generic.go:356] "Generic (PLEG): container finished" podID="4fa26111-5767-466d-94f7-cb2b5e79690a" containerID="b7db9e3ae04baaaf91e45b3701bcaf191fe3b2bc24f7efd547b06ed167c7e659" exitCode=0
Nov 03 20:39:17.441969 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:17.441581    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5" event={"ID":"4fa26111-5767-466d-94f7-cb2b5e79690a","Type":"ContainerDied","Data":"b7db9e3ae04baaaf91e45b3701bcaf191fe3b2bc24f7efd547b06ed167c7e659"}
Nov 03 20:39:18.587658 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:18.587629    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:18.728181 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:18.728110    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa26111-5767-466d-94f7-cb2b5e79690a-kube-api-access-h5xh9\" (UniqueName: \"kubernetes.io/projected/4fa26111-5767-466d-94f7-cb2b5e79690a-kube-api-access-h5xh9\") pod \"4fa26111-5767-466d-94f7-cb2b5e79690a\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") "
Nov 03 20:39:18.728314 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:18.728225    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-xks-ca\") pod \"4fa26111-5767-466d-94f7-cb2b5e79690a\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") "
Nov 03 20:39:18.728314 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:18.728259    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-openshift-ca\") pod \"4fa26111-5767-466d-94f7-cb2b5e79690a\" (UID: \"4fa26111-5767-466d-94f7-cb2b5e79690a\") "
Nov 03 20:39:18.728633 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:18.728610    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-openshift-ca" pod "4fa26111-5767-466d-94f7-cb2b5e79690a" (UID: "4fa26111-5767-466d-94f7-cb2b5e79690a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:18.730410 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:18.730384    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa26111-5767-466d-94f7-cb2b5e79690a-kube-api-access-h5xh9" pod "4fa26111-5767-466d-94f7-cb2b5e79690a" (UID: "4fa26111-5767-466d-94f7-cb2b5e79690a"). InnerVolumeSpecName "kube-api-access-h5xh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:18.730846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:18.730583    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-xks-ca" pod "4fa26111-5767-466d-94f7-cb2b5e79690a" (UID: "4fa26111-5767-466d-94f7-cb2b5e79690a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:18.829321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:18.829280    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5xh9\" (UniqueName: \"kubernetes.io/projected/4fa26111-5767-466d-94f7-cb2b5e79690a-kube-api-access-h5xh9\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:18.829321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:18.829323    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:18.829559 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:18.829338    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa26111-5767-466d-94f7-cb2b5e79690a-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:19.461782 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:19.461749    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"
Nov 03 20:39:19.461782 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:19.461759    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5" event={"ID":"4fa26111-5767-466d-94f7-cb2b5e79690a","Type":"ContainerDied","Data":"1f61bf38423e73ade8d793b7ac02713ce053c52e522cb4389065c82ea89e1c56"}
Nov 03 20:39:19.462012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:19.461829    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f61bf38423e73ade8d793b7ac02713ce053c52e522cb4389065c82ea89e1c56"
Nov 03 20:39:20.383957 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:20.383923    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"]
Nov 03 20:39:22.285128 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:22.285093    5600 scope.go:122] "RemoveContainer" containerID="f49be9b53883ee2ccb18a0f3549f449d48173aca7866975c62dc3b71f10cbb21"
Nov 03 20:39:27.090088 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:27.090050    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"]
Nov 03 20:39:27.092588 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:27.092564    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4442a4-7dth5"]
Nov 03 20:39:27.616248 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:27.616215    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa26111-5767-466d-94f7-cb2b5e79690a" path="/var/lib/kubelet/pods/4fa26111-5767-466d-94f7-cb2b5e79690a/volumes"
Nov 03 20:39:28.794752 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:39:28.794713    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:39:35.410795 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.410738    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf" podUID="f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" containerName="main" containerID="cri-o://47c89a80ca400a6d66daa06b639be8bcaa74959f08f6603657fa5b29fecd3c4a" gracePeriod=45
Nov 03 20:39:35.548106 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.548075    5600 generic.go:356] "Generic (PLEG): container finished" podID="f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" containerID="47c89a80ca400a6d66daa06b639be8bcaa74959f08f6603657fa5b29fecd3c4a" exitCode=0
Nov 03 20:39:35.548253 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.548150    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf" event={"ID":"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0","Type":"ContainerDied","Data":"47c89a80ca400a6d66daa06b639be8bcaa74959f08f6603657fa5b29fecd3c4a"}
Nov 03 20:39:35.658757 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.658735    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:39:35.682785 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.682705    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-home\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-home\") pod \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") "
Nov 03 20:39:35.682939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.682796    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-dshm\") pod \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") "
Nov 03 20:39:35.682939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.682848    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-kube-api-access-j4gl9\" (UniqueName: \"kubernetes.io/projected/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-kube-api-access-j4gl9\") pod \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") "
Nov 03 20:39:35.682939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.682897    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tmp-dir\") pod \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") "
Nov 03 20:39:35.682939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.682928    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-model-cache\") pod \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") "
Nov 03 20:39:35.683183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.682949    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tls-certs\") pod \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\" (UID: \"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0\") "
Nov 03 20:39:35.683183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.683046    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-home" pod "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" (UID: "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:35.683308 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.683274    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tmp-dir" pod "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" (UID: "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:35.683966 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.683694    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.684259 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.684242    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.684462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.683774    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-model-cache" pod "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" (UID: "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:35.686313 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.686283    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-kube-api-access-j4gl9" pod "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" (UID: "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0"). InnerVolumeSpecName "kube-api-access-j4gl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:35.686420 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.686349    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-dshm" pod "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" (UID: "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:35.687665 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.687639    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tls-certs" pod "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" (UID: "f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:35.785633 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.785603    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.785633 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.785629    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4gl9\" (UniqueName: \"kubernetes.io/projected/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-kube-api-access-j4gl9\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.785633 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.785640    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.785870 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:35.785650    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:36.554639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:36.554606    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"
Nov 03 20:39:36.555158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:36.554641    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf" event={"ID":"f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0","Type":"ContainerDied","Data":"24bb70489ae9775202b0133b207a7ae08d9ca18c0181563fba0ff8dd3f36b73d"}
Nov 03 20:39:36.555158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:36.554701    5600 scope.go:122] "RemoveContainer" containerID="47c89a80ca400a6d66daa06b639be8bcaa74959f08f6603657fa5b29fecd3c4a"
Nov 03 20:39:36.575316 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:36.575293    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"]
Nov 03 20:39:36.579317 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:36.579285    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-4442a4/simulator-single-main-4442a4-kserve-74f89ff7c7-k6nkf"]
Nov 03 20:39:37.614846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:37.614814    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" path="/var/lib/kubelet/pods/f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0/volumes"
Nov 03 20:39:58.720740 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:58.720697    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"]
Nov 03 20:39:58.721245 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:58.720952    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t" podUID="85077442-78c8-48dc-85f1-ce4bb58bd59c" containerName="manager" containerID="cri-o://d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b" gracePeriod=10
Nov 03 20:39:58.793914 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:39:58.793881    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:39:58.974777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:58.974723    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 20:39:59.113779 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.113742    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85077442-78c8-48dc-85f1-ce4bb58bd59c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85077442-78c8-48dc-85f1-ce4bb58bd59c-extensions-socket-volume\") pod \"85077442-78c8-48dc-85f1-ce4bb58bd59c\" (UID: \"85077442-78c8-48dc-85f1-ce4bb58bd59c\") "
Nov 03 20:39:59.113984 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.113803    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85077442-78c8-48dc-85f1-ce4bb58bd59c-kube-api-access-rwsw7\" (UniqueName: \"kubernetes.io/projected/85077442-78c8-48dc-85f1-ce4bb58bd59c-kube-api-access-rwsw7\") pod \"85077442-78c8-48dc-85f1-ce4bb58bd59c\" (UID: \"85077442-78c8-48dc-85f1-ce4bb58bd59c\") "
Nov 03 20:39:59.114231 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.114203    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85077442-78c8-48dc-85f1-ce4bb58bd59c-extensions-socket-volume" pod "85077442-78c8-48dc-85f1-ce4bb58bd59c" (UID: "85077442-78c8-48dc-85f1-ce4bb58bd59c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:59.116018 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.115994    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85077442-78c8-48dc-85f1-ce4bb58bd59c-kube-api-access-rwsw7" pod "85077442-78c8-48dc-85f1-ce4bb58bd59c" (UID: "85077442-78c8-48dc-85f1-ce4bb58bd59c"). InnerVolumeSpecName "kube-api-access-rwsw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:59.215060 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.215022    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwsw7\" (UniqueName: \"kubernetes.io/projected/85077442-78c8-48dc-85f1-ce4bb58bd59c-kube-api-access-rwsw7\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:59.215060 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.215055    5600 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85077442-78c8-48dc-85f1-ce4bb58bd59c-extensions-socket-volume\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:59.679906 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.679874    5600 generic.go:356] "Generic (PLEG): container finished" podID="85077442-78c8-48dc-85f1-ce4bb58bd59c" containerID="d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b" exitCode=0
Nov 03 20:39:59.680072 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.679934    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"
Nov 03 20:39:59.680072 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.679966    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t" event={"ID":"85077442-78c8-48dc-85f1-ce4bb58bd59c","Type":"ContainerDied","Data":"d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b"}
Nov 03 20:39:59.680072 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.680019    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t" event={"ID":"85077442-78c8-48dc-85f1-ce4bb58bd59c","Type":"ContainerDied","Data":"afbef16fb943ff7c51d188f5c8cdee4334306b159f55f3ff6bd4aa56c285217d"}
Nov 03 20:39:59.680072 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.680040    5600 scope.go:122] "RemoveContainer" containerID="d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b"
Nov 03 20:39:59.692939 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.692919    5600 scope.go:122] "RemoveContainer" containerID="d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b"
Nov 03 20:39:59.693225 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:39:59.693206    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b\": container with ID starting with d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b not found: ID does not exist" containerID="d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b"
Nov 03 20:39:59.693274 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.693234    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b"} err="failed to get container status \"d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b\": rpc error: code = NotFound desc = could not find container \"d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b\": container with ID starting with d5542b5e0b0f4400ae53f9be98538ca790603d17c012d165e70339a3cdc39f6b not found: ID does not exist"
Nov 03 20:39:59.699418 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.699397    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"]
Nov 03 20:39:59.704956 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:39:59.704936    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7v42t"]
Nov 03 20:40:01.615848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:01.615815    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85077442-78c8-48dc-85f1-ce4bb58bd59c" path="/var/lib/kubelet/pods/85077442-78c8-48dc-85f1-ce4bb58bd59c/volumes"
Nov 03 20:40:03.598528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.598494    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"]
Nov 03 20:40:03.598952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.598887    5600 cpu_manager.go:414] "containerMap: removing container" podUID="4fa26111-5767-466d-94f7-cb2b5e79690a" containerName="revoke-keys"
Nov 03 20:40:03.598952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.598900    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa26111-5767-466d-94f7-cb2b5e79690a" containerName="revoke-keys"
Nov 03 20:40:03.598952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.598932    5600 cpu_manager.go:414] "containerMap: removing container" podUID="85077442-78c8-48dc-85f1-ce4bb58bd59c" containerName="manager"
Nov 03 20:40:03.598952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.598936    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85077442-78c8-48dc-85f1-ce4bb58bd59c" containerName="manager"
Nov 03 20:40:03.598952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.598950    5600 cpu_manager.go:414] "containerMap: removing container" podUID="f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" containerName="main"
Nov 03 20:40:03.598952 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.598956    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" containerName="main"
Nov 03 20:40:03.599183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.599024    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="85077442-78c8-48dc-85f1-ce4bb58bd59c" containerName="manager"
Nov 03 20:40:03.599183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.599032    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fd8f08-a3ab-4bde-ac57-f77cf2b5f0d0" containerName="main"
Nov 03 20:40:03.599183 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.599041    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa26111-5767-466d-94f7-cb2b5e79690a" containerName="revoke-keys"
Nov 03 20:40:03.603679 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.603645    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.616965 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.616941    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"]
Nov 03 20:40:03.757383 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.757334    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.757383 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.757384    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.757688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.757529    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.757688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.757590    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.757688 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.757624    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.757818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.757702    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g4pv\" (UniqueName: \"kubernetes.io/projected/c52f10e7-447a-4942-ba2e-e42509bf6e33-kube-api-access-7g4pv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.757818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.757741    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.757818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.757762    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.757818 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.757778    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52f10e7-447a-4942-ba2e-e42509bf6e33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.858827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.858727    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.858827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.858775    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52f10e7-447a-4942-ba2e-e42509bf6e33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.858827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.858803    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.858827 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.858821    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.859144 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.858870    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.859144 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.858901    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.859144 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.858932    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.859144 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.858973    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g4pv\" (UniqueName: \"kubernetes.io/projected/c52f10e7-447a-4942-ba2e-e42509bf6e33-kube-api-access-7g4pv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.859144 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.859018    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.859419 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.859307    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.859505 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.859421    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.859561 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.859497    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.859657 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.859639    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52f10e7-447a-4942-ba2e-e42509bf6e33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.859695 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.859657    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.861833 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.861816    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.862156 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.862125    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.866836 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.866810    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52f10e7-447a-4942-ba2e-e42509bf6e33-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.867259 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.867237    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g4pv\" (UniqueName: \"kubernetes.io/projected/c52f10e7-447a-4942-ba2e-e42509bf6e33-kube-api-access-7g4pv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-wp8sz\" (UID: \"c52f10e7-447a-4942-ba2e-e42509bf6e33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:03.920655 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:03.920625    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:04.196902 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:04.196875    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"]
Nov 03 20:40:04.198880 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:40:04.198848    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52f10e7_447a_4942_ba2e_e42509bf6e33.slice/crio-0345ef05b2e38f94bac462f2d2bf3e9aaead8a1bf7d2e0a904713fb82bb097c4 WatchSource:0}: Error finding container 0345ef05b2e38f94bac462f2d2bf3e9aaead8a1bf7d2e0a904713fb82bb097c4: Status 404 returned error can't find the container with id 0345ef05b2e38f94bac462f2d2bf3e9aaead8a1bf7d2e0a904713fb82bb097c4
Nov 03 20:40:04.201161 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:04.201128    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:40:04.201241 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:04.201207    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:40:04.201280 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:04.201246    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:40:04.710148 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:04.710094    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz" event={"ID":"c52f10e7-447a-4942-ba2e-e42509bf6e33","Type":"ContainerStarted","Data":"aa7caf9bad97eba2b0e71f40bdb0fa7644a49fd1502f718843c136f771d73831"}
Nov 03 20:40:04.710661 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:04.710233    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz" event={"ID":"c52f10e7-447a-4942-ba2e-e42509bf6e33","Type":"ContainerStarted","Data":"0345ef05b2e38f94bac462f2d2bf3e9aaead8a1bf7d2e0a904713fb82bb097c4"}
Nov 03 20:40:04.733227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:04.733176    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz" podStartSLOduration=1.733161628 podStartE2EDuration="1.733161628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:04.732236495 +0000 UTC m=+3587.245058270" watchObservedRunningTime="2026-11-03 20:40:04.733161628 +0000 UTC m=+3587.245983407"
Nov 03 20:40:04.920731 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:04.920698    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:05.926213 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:05.926166    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:06.720933 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:06.720905    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:06.721846 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:06.721828    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:40:18.603368 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.603330    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"]
Nov 03 20:40:18.607202 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.607181    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.616674 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.616647    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"]
Nov 03 20:40:18.688714 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.688686    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.688882 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.688725    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.688882 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.688747    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.688882 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.688805    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.688990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.688920    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.688990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.688966    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.689061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.688988    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.689061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.689021    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.689126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.689061    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltwp\" (UniqueName: \"kubernetes.io/projected/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-kube-api-access-7ltwp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.789852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.789819    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.789874    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.789936    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.789955    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.789972    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltwp\" (UniqueName: \"kubernetes.io/projected/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-kube-api-access-7ltwp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.790028    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.790046    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.790064    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.790087    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790401 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.790384    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790615 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.790587    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790666 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.790598    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.790847 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.790822    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.791220 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.791198    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.793001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.792978    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.793112 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.793096    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.798753 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.798727    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.798970 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.798953    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltwp\" (UniqueName: \"kubernetes.io/projected/770a59d3-9646-4f7a-a152-3eeaae4cbfbf-kube-api-access-7ltwp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-k6fhj\" (UID: \"770a59d3-9646-4f7a-a152-3eeaae4cbfbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:18.923261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:18.923227    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:19.200692 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:19.200662    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"]
Nov 03 20:40:19.202797 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:40:19.202762    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770a59d3_9646_4f7a_a152_3eeaae4cbfbf.slice/crio-c2e196da224f6a50843e9a01a39af7e82e73c392d646df49ba7aa5af4d78906a WatchSource:0}: Error finding container c2e196da224f6a50843e9a01a39af7e82e73c392d646df49ba7aa5af4d78906a: Status 404 returned error can't find the container with id c2e196da224f6a50843e9a01a39af7e82e73c392d646df49ba7aa5af4d78906a
Nov 03 20:40:19.205403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:19.205376    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:40:19.205489 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:19.205464    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:40:19.205531 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:19.205513    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:40:19.799326 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:19.799278    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" event={"ID":"770a59d3-9646-4f7a-a152-3eeaae4cbfbf","Type":"ContainerStarted","Data":"3aa261e278b7ffbfb3a46b407444c48c0b5ee1359a88babba3d1282380bb5c11"}
Nov 03 20:40:19.799835 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:19.799351    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" event={"ID":"770a59d3-9646-4f7a-a152-3eeaae4cbfbf","Type":"ContainerStarted","Data":"c2e196da224f6a50843e9a01a39af7e82e73c392d646df49ba7aa5af4d78906a"}
Nov 03 20:40:19.822337 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:19.822279    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" podStartSLOduration=1.8222639699999998 podStartE2EDuration="1.82226397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:19.82085902 +0000 UTC m=+3602.333680800" watchObservedRunningTime="2026-11-03 20:40:19.82226397 +0000 UTC m=+3602.335085752"
Nov 03 20:40:19.923351 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:19.923319    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:20.929717 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:20.929690    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:21.809930 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:21.809898    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:21.810884 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:40:21.810861    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:40:28.794575 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:40:28.794494    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:40:58.793846 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:40:58.793804    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:41:10.098674 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.098638    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"]
Nov 03 20:41:10.102402 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.102384    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"
Nov 03 20:41:10.115296 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.115257    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"]
Nov 03 20:41:10.146009 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.145980    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d5jrn\""
Nov 03 20:41:10.175378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.175351    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53f8ba01-60ee-4a82-880e-2fcf8ba98d68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5dvbz\" (UID: \"53f8ba01-60ee-4a82-880e-2fcf8ba98d68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"
Nov 03 20:41:10.175548 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.175413    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jntfp\" (UniqueName: \"kubernetes.io/projected/53f8ba01-60ee-4a82-880e-2fcf8ba98d68-kube-api-access-jntfp\") pod \"kuadrant-operator-controller-manager-678d87766f-5dvbz\" (UID: \"53f8ba01-60ee-4a82-880e-2fcf8ba98d68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"
Nov 03 20:41:10.276513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.276480    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jntfp\" (UniqueName: \"kubernetes.io/projected/53f8ba01-60ee-4a82-880e-2fcf8ba98d68-kube-api-access-jntfp\") pod \"kuadrant-operator-controller-manager-678d87766f-5dvbz\" (UID: \"53f8ba01-60ee-4a82-880e-2fcf8ba98d68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"
Nov 03 20:41:10.276682 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.276575    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53f8ba01-60ee-4a82-880e-2fcf8ba98d68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5dvbz\" (UID: \"53f8ba01-60ee-4a82-880e-2fcf8ba98d68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"
Nov 03 20:41:10.276934 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.276913    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53f8ba01-60ee-4a82-880e-2fcf8ba98d68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5dvbz\" (UID: \"53f8ba01-60ee-4a82-880e-2fcf8ba98d68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"
Nov 03 20:41:10.285077 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.285043    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jntfp\" (UniqueName: \"kubernetes.io/projected/53f8ba01-60ee-4a82-880e-2fcf8ba98d68-kube-api-access-jntfp\") pod \"kuadrant-operator-controller-manager-678d87766f-5dvbz\" (UID: \"53f8ba01-60ee-4a82-880e-2fcf8ba98d68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"
Nov 03 20:41:10.416884 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.416850    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"
Nov 03 20:41:10.707564 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:10.707537    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"]
Nov 03 20:41:10.708721 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:41:10.708695    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f8ba01_60ee_4a82_880e_2fcf8ba98d68.slice/crio-e6550a4b6e572fd7f95a6e00be0fc192080387108e2572f7149f58b561512735 WatchSource:0}: Error finding container e6550a4b6e572fd7f95a6e00be0fc192080387108e2572f7149f58b561512735: Status 404 returned error can't find the container with id e6550a4b6e572fd7f95a6e00be0fc192080387108e2572f7149f58b561512735
Nov 03 20:41:11.054908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:11.054807    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz" event={"ID":"53f8ba01-60ee-4a82-880e-2fcf8ba98d68","Type":"ContainerStarted","Data":"6f454df86409136a2c994a8669baf75e997518dd51d5d1dd73a5dc6e6ab6b585"}
Nov 03 20:41:11.054908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:11.054863    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz" event={"ID":"53f8ba01-60ee-4a82-880e-2fcf8ba98d68","Type":"ContainerStarted","Data":"e6550a4b6e572fd7f95a6e00be0fc192080387108e2572f7149f58b561512735"}
Nov 03 20:41:11.054908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:11.054886    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"
Nov 03 20:41:11.080008 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:11.079960    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz" podStartSLOduration=1.079948504 podStartE2EDuration="1.079948504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:11.078968547 +0000 UTC m=+3653.591790329" watchObservedRunningTime="2026-11-03 20:41:11.079948504 +0000 UTC m=+3653.592770284"
Nov 03 20:41:22.061968 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:22.061936    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5dvbz"
Nov 03 20:41:28.805002 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:41:28.804961    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:41:37.852641 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:37.852594    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k6fhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:41:37.853167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:37.852680    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" podUID="770a59d3-9646-4f7a-a152-3eeaae4cbfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:41:40.207780 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.207744    5600 generic.go:356] "Generic (PLEG): container finished" podID="c52f10e7-447a-4942-ba2e-e42509bf6e33" containerID="aa7caf9bad97eba2b0e71f40bdb0fa7644a49fd1502f718843c136f771d73831" exitCode=0
Nov 03 20:41:40.208327 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.207823    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz" event={"ID":"c52f10e7-447a-4942-ba2e-e42509bf6e33","Type":"ContainerDied","Data":"aa7caf9bad97eba2b0e71f40bdb0fa7644a49fd1502f718843c136f771d73831"}
Nov 03 20:41:40.208327 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.208240    5600 scope.go:122] "RemoveContainer" containerID="aa7caf9bad97eba2b0e71f40bdb0fa7644a49fd1502f718843c136f771d73831"
Nov 03 20:41:40.209508 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.209476    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:41:40.209651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.209536    5600 generic.go:356] "Generic (PLEG): container finished" podID="770a59d3-9646-4f7a-a152-3eeaae4cbfbf" containerID="3aa261e278b7ffbfb3a46b407444c48c0b5ee1359a88babba3d1282380bb5c11" exitCode=0
Nov 03 20:41:40.209651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.209563    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:41:40.209651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.209610    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:41:40.209651 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.209609    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" event={"ID":"770a59d3-9646-4f7a-a152-3eeaae4cbfbf","Type":"ContainerDied","Data":"3aa261e278b7ffbfb3a46b407444c48c0b5ee1359a88babba3d1282380bb5c11"}
Nov 03 20:41:40.209920 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.209903    5600 scope.go:122] "RemoveContainer" containerID="3aa261e278b7ffbfb3a46b407444c48c0b5ee1359a88babba3d1282380bb5c11"
Nov 03 20:41:40.210889 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.210865    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:41:40.210985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.210933    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:41:40.210985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:40.210977    5600 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 20:41:41.235758 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:41.235711    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz" event={"ID":"c52f10e7-447a-4942-ba2e-e42509bf6e33","Type":"ContainerStarted","Data":"a1f65b048b2a1e12e0805847f8df55898574cae94356b4690b083f6d53992411"}
Nov 03 20:41:41.239811 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:41.239785    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" event={"ID":"770a59d3-9646-4f7a-a152-3eeaae4cbfbf","Type":"ContainerStarted","Data":"2f52cf4cf56b5e4f2703c1ac0b72f701ca4bcbdb44e0b14494e489dad4ac25e3"}
Nov 03 20:41:41.922458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:41.921840    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:41:41.922458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:41.921972    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-wp8sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:41:41.922458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:41.922018    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz" podUID="c52f10e7-447a-4942-ba2e-e42509bf6e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 20:41:41.929462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:41.925760    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:41:41.929462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:41.926239    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k6fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:41:41.929462 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:41.926288    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" podUID="770a59d3-9646-4f7a-a152-3eeaae4cbfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 20:41:42.926364 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:42.925886    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-wp8sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:41:42.926364 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:42.925947    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz" podUID="c52f10e7-447a-4942-ba2e-e42509bf6e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 20:41:42.927423 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:42.927320    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k6fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:41:42.927423 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:42.927372    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" podUID="770a59d3-9646-4f7a-a152-3eeaae4cbfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 20:41:43.929461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:43.925870    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-wp8sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:41:43.929461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:43.925936    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz" podUID="c52f10e7-447a-4942-ba2e-e42509bf6e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 20:41:43.929461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:43.926675    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k6fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:41:43.929461 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:43.926716    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" podUID="770a59d3-9646-4f7a-a152-3eeaae4cbfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 20:41:44.924944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:44.924902    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-wp8sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:41:44.926382 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:44.926347    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz" podUID="c52f10e7-447a-4942-ba2e-e42509bf6e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 20:41:44.926618 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:44.925758    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k6fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:41:44.926763 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:44.926743    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" podUID="770a59d3-9646-4f7a-a152-3eeaae4cbfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 20:41:45.921994 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:45.921949    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-wp8sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:41:45.922723 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:45.922681    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz" podUID="c52f10e7-447a-4942-ba2e-e42509bf6e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 20:41:45.927575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:45.927544    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k6fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:41:45.927775 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:45.927752    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" podUID="770a59d3-9646-4f7a-a152-3eeaae4cbfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 20:41:46.921210 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:46.921164    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-wp8sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:41:46.921474 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:46.921249    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz" podUID="c52f10e7-447a-4942-ba2e-e42509bf6e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 20:41:46.924515 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:46.924485    5600 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-k6fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:41:46.924981 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:46.924529    5600 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj" podUID="770a59d3-9646-4f7a-a152-3eeaae4cbfbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 20:41:48.523985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:48.522826    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:41:48.523985 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:48.523929    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:41:48.524597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:48.524575    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-k6fhj"
Nov 03 20:41:48.549560 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:48.549518    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:41:48.550017 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:48.549956    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:41:48.551375 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:41:48.551349    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-wp8sz"
Nov 03 20:41:58.863561 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:41:58.863411    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:42:28.795340 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:42:28.795303    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:42:35.794672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.794634    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"]
Nov 03 20:42:35.799031 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.798995    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:35.807116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.807073    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4b0085\"/\"simulator-single-main-4b0085-kserve-self-signed-certs\""
Nov 03 20:42:35.807486 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.807448    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4b0085\"/\"default-dockercfg-n7nx6\""
Nov 03 20:42:35.807749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.807718    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4b0085\"/\"kube-root-ca.crt\""
Nov 03 20:42:35.807860 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.807834    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4b0085\"/\"openshift-service-ca.crt\""
Nov 03 20:42:35.809811 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.809783    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"]
Nov 03 20:42:35.907899 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.907861    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-tmp-dir\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:35.908103 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.907906    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-dshm\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:35.908103 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.907962    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-home\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:35.908103 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.908054    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-model-cache\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:35.908214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.908102    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6c7d443-695f-4d43-9293-33160e77157f-tls-certs\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:35.908214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:35.908207    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4v5h\" (UniqueName: \"kubernetes.io/projected/f6c7d443-695f-4d43-9293-33160e77157f-kube-api-access-w4v5h\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.009629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.009587    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4v5h\" (UniqueName: \"kubernetes.io/projected/f6c7d443-695f-4d43-9293-33160e77157f-kube-api-access-w4v5h\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.009852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.009651    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-tmp-dir\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.009852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.009675    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-dshm\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.009852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.009705    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-home\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.009852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.009734    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-model-cache\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.009852 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.009794    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6c7d443-695f-4d43-9293-33160e77157f-tls-certs\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.010204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.010069    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-tmp-dir\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.010204 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.010092    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-home\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.010308 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.010262    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-model-cache\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.012247 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.012220    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-dshm\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.012600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.012583    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6c7d443-695f-4d43-9293-33160e77157f-tls-certs\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.017764 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.017737    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4v5h\" (UniqueName: \"kubernetes.io/projected/f6c7d443-695f-4d43-9293-33160e77157f-kube-api-access-w4v5h\") pod \"simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") " pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.114687 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.114595    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:36.396777 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.396737    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"]
Nov 03 20:42:36.399803 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:42:36.399770    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6c7d443_695f_4d43_9293_33160e77157f.slice/crio-25046d96ad908ec87168cf5ea5b57383555b91b806b42c9b9c7f142829aad8ad WatchSource:0}: Error finding container 25046d96ad908ec87168cf5ea5b57383555b91b806b42c9b9c7f142829aad8ad: Status 404 returned error can't find the container with id 25046d96ad908ec87168cf5ea5b57383555b91b806b42c9b9c7f142829aad8ad
Nov 03 20:42:36.571380 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.571342    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp" event={"ID":"f6c7d443-695f-4d43-9293-33160e77157f","Type":"ContainerStarted","Data":"33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437"}
Nov 03 20:42:36.571600 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.571396    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp" event={"ID":"f6c7d443-695f-4d43-9293-33160e77157f","Type":"ContainerStarted","Data":"25046d96ad908ec87168cf5ea5b57383555b91b806b42c9b9c7f142829aad8ad"}
Nov 03 20:42:36.594664 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:36.594599    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp" podStartSLOduration=1.594583944 podStartE2EDuration="1.594583944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:36.592315127 +0000 UTC m=+3739.105136915" watchObservedRunningTime="2026-11-03 20:42:36.594583944 +0000 UTC m=+3739.107405789"
Nov 03 20:42:37.115236 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:37.115197    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:46.115513 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:46.115475    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:46.127900 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:46.127872    5600 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:46.633467 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:42:46.633426    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:42:58.794715 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:42:58.794667    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:43:04.790321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.790289    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"]
Nov 03 20:43:04.794012 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.793993    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:04.799095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.799064    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:43:04.802574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.800331    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:43:04.803290 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.803136    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"]
Nov 03 20:43:04.884065 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.884021    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gspcc\" (UniqueName: \"kubernetes.io/projected/9d63a767-842d-4ebd-8e26-b613e5dc33ba-kube-api-access-gspcc\") pod \"maas-api-revoke-sub-5bed252c-dd6zt\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:04.884241 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.884165    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bed252c-dd6zt\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:04.884293 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.884242    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bed252c-dd6zt\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:04.985272 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.985229    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bed252c-dd6zt\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:04.985500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.985288    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bed252c-dd6zt\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:04.985500 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.985331    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gspcc\" (UniqueName: \"kubernetes.io/projected/9d63a767-842d-4ebd-8e26-b613e5dc33ba-kube-api-access-gspcc\") pod \"maas-api-revoke-sub-5bed252c-dd6zt\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:04.986109 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.986076    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bed252c-dd6zt\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:04.988126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.988096    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bed252c-dd6zt\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:04.993472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:04.993423    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gspcc\" (UniqueName: \"kubernetes.io/projected/9d63a767-842d-4ebd-8e26-b613e5dc33ba-kube-api-access-gspcc\") pod \"maas-api-revoke-sub-5bed252c-dd6zt\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:05.111670 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:05.111587    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:05.388727 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:05.388696    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"]
Nov 03 20:43:05.389816 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:43:05.389786    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d63a767_842d_4ebd_8e26_b613e5dc33ba.slice/crio-67558de16d45ad8f6466e24b81b712c9b2e1e3a428e4df93f223a0ddf08dd62c WatchSource:0}: Error finding container 67558de16d45ad8f6466e24b81b712c9b2e1e3a428e4df93f223a0ddf08dd62c: Status 404 returned error can't find the container with id 67558de16d45ad8f6466e24b81b712c9b2e1e3a428e4df93f223a0ddf08dd62c
Nov 03 20:43:05.729138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:05.729093    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt" event={"ID":"9d63a767-842d-4ebd-8e26-b613e5dc33ba","Type":"ContainerStarted","Data":"3f8829499f463bd0a387f90d8ece1e73972dcca0be72fe8e2e9d80fa43bdcfcd"}
Nov 03 20:43:05.729295 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:05.729168    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt" event={"ID":"9d63a767-842d-4ebd-8e26-b613e5dc33ba","Type":"ContainerStarted","Data":"67558de16d45ad8f6466e24b81b712c9b2e1e3a428e4df93f223a0ddf08dd62c"}
Nov 03 20:43:05.748125 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:05.748071    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt" podStartSLOduration=1.748056941 podStartE2EDuration="1.748056941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:05.746790165 +0000 UTC m=+3768.259611947" watchObservedRunningTime="2026-11-03 20:43:05.748056941 +0000 UTC m=+3768.260878723"
Nov 03 20:43:06.736546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:06.736511    5600 generic.go:356] "Generic (PLEG): container finished" podID="9d63a767-842d-4ebd-8e26-b613e5dc33ba" containerID="3f8829499f463bd0a387f90d8ece1e73972dcca0be72fe8e2e9d80fa43bdcfcd" exitCode=0
Nov 03 20:43:06.737158 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:06.736584    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt" event={"ID":"9d63a767-842d-4ebd-8e26-b613e5dc33ba","Type":"ContainerDied","Data":"3f8829499f463bd0a387f90d8ece1e73972dcca0be72fe8e2e9d80fa43bdcfcd"}
Nov 03 20:43:07.891629 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:07.891601    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:08.019040 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.018956    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-openshift-ca\") pod \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") "
Nov 03 20:43:08.019193 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.019113    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d63a767-842d-4ebd-8e26-b613e5dc33ba-kube-api-access-gspcc\" (UniqueName: \"kubernetes.io/projected/9d63a767-842d-4ebd-8e26-b613e5dc33ba-kube-api-access-gspcc\") pod \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") "
Nov 03 20:43:08.019193 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.019186    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-xks-ca\") pod \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\" (UID: \"9d63a767-842d-4ebd-8e26-b613e5dc33ba\") "
Nov 03 20:43:08.019399 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.019378    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-openshift-ca" pod "9d63a767-842d-4ebd-8e26-b613e5dc33ba" (UID: "9d63a767-842d-4ebd-8e26-b613e5dc33ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:43:08.019528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.019513    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:08.021449 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.021401    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-xks-ca" pod "9d63a767-842d-4ebd-8e26-b613e5dc33ba" (UID: "9d63a767-842d-4ebd-8e26-b613e5dc33ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:43:08.021582 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.021561    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d63a767-842d-4ebd-8e26-b613e5dc33ba-kube-api-access-gspcc" pod "9d63a767-842d-4ebd-8e26-b613e5dc33ba" (UID: "9d63a767-842d-4ebd-8e26-b613e5dc33ba"). InnerVolumeSpecName "kube-api-access-gspcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:43:08.120656 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.120602    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d63a767-842d-4ebd-8e26-b613e5dc33ba-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:08.120656 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.120659    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gspcc\" (UniqueName: \"kubernetes.io/projected/9d63a767-842d-4ebd-8e26-b613e5dc33ba-kube-api-access-gspcc\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:08.749153 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.749118    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt" event={"ID":"9d63a767-842d-4ebd-8e26-b613e5dc33ba","Type":"ContainerDied","Data":"67558de16d45ad8f6466e24b81b712c9b2e1e3a428e4df93f223a0ddf08dd62c"}
Nov 03 20:43:08.749153 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.749149    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"
Nov 03 20:43:08.749378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:08.749167    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67558de16d45ad8f6466e24b81b712c9b2e1e3a428e4df93f223a0ddf08dd62c"
Nov 03 20:43:15.730669 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.730633    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"]
Nov 03 20:43:15.731257 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.731235    5600 cpu_manager.go:414] "containerMap: removing container" podUID="9d63a767-842d-4ebd-8e26-b613e5dc33ba" containerName="revoke-keys"
Nov 03 20:43:15.731345 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.731260    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d63a767-842d-4ebd-8e26-b613e5dc33ba" containerName="revoke-keys"
Nov 03 20:43:15.731403 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.731361    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d63a767-842d-4ebd-8e26-b613e5dc33ba" containerName="revoke-keys"
Nov 03 20:43:15.734790 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.734767    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:15.740810 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.740783    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:43:15.741126 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.741088    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:43:15.744148 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.744127    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"]
Nov 03 20:43:15.894249 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.894210    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:15.894477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.894341    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:15.894477 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.894376    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4wnf\" (UniqueName: \"kubernetes.io/projected/2cf82758-35f5-4134-b0cf-5b92a9ed951e-kube-api-access-d4wnf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:15.996001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.995918    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:15.996001 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.995993    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:15.996214 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.996021    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wnf\" (UniqueName: \"kubernetes.io/projected/2cf82758-35f5-4134-b0cf-5b92a9ed951e-kube-api-access-d4wnf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:15.996783 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.996755    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:15.998594 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:15.998575    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:16.006451 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:16.006407    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4wnf\" (UniqueName: \"kubernetes.io/projected/2cf82758-35f5-4134-b0cf-5b92a9ed951e-kube-api-access-d4wnf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:16.051041 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:16.051004    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:16.347597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:16.347572    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"]
Nov 03 20:43:16.349411 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:43:16.349384    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf82758_35f5_4134_b0cf_5b92a9ed951e.slice/crio-4de19a0a71d65a98614a93afb32ab76da211a1a20050a27558a8dba71d259ccb WatchSource:0}: Error finding container 4de19a0a71d65a98614a93afb32ab76da211a1a20050a27558a8dba71d259ccb: Status 404 returned error can't find the container with id 4de19a0a71d65a98614a93afb32ab76da211a1a20050a27558a8dba71d259ccb
Nov 03 20:43:16.800865 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:16.800827    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4" event={"ID":"2cf82758-35f5-4134-b0cf-5b92a9ed951e","Type":"ContainerStarted","Data":"0f2aa6a65ec458b47163180acb00f4ef849bf925263cf466fa5393153c07ed70"}
Nov 03 20:43:16.801346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:16.800885    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4" event={"ID":"2cf82758-35f5-4134-b0cf-5b92a9ed951e","Type":"ContainerStarted","Data":"4de19a0a71d65a98614a93afb32ab76da211a1a20050a27558a8dba71d259ccb"}
Nov 03 20:43:16.818597 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:16.817928    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4" podStartSLOduration=1.817910684 podStartE2EDuration="1.817910684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:16.816774962 +0000 UTC m=+3779.329596748" watchObservedRunningTime="2026-11-03 20:43:16.817910684 +0000 UTC m=+3779.330732466"
Nov 03 20:43:17.808004 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:17.807963    5600 generic.go:356] "Generic (PLEG): container finished" podID="2cf82758-35f5-4134-b0cf-5b92a9ed951e" containerID="0f2aa6a65ec458b47163180acb00f4ef849bf925263cf466fa5393153c07ed70" exitCode=0
Nov 03 20:43:17.808409 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:17.808010    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4" event={"ID":"2cf82758-35f5-4134-b0cf-5b92a9ed951e","Type":"ContainerDied","Data":"0f2aa6a65ec458b47163180acb00f4ef849bf925263cf466fa5393153c07ed70"}
Nov 03 20:43:18.951708 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:18.951684    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:19.025305 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.025271    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-openshift-ca\") pod \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") "
Nov 03 20:43:19.025305 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.025310    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-xks-ca\") pod \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") "
Nov 03 20:43:19.025623 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.025509    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf82758-35f5-4134-b0cf-5b92a9ed951e-kube-api-access-d4wnf\" (UniqueName: \"kubernetes.io/projected/2cf82758-35f5-4134-b0cf-5b92a9ed951e-kube-api-access-d4wnf\") pod \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\" (UID: \"2cf82758-35f5-4134-b0cf-5b92a9ed951e\") "
Nov 03 20:43:19.025757 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.025731    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-openshift-ca" pod "2cf82758-35f5-4134-b0cf-5b92a9ed951e" (UID: "2cf82758-35f5-4134-b0cf-5b92a9ed951e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:43:19.025875 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.025859    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:19.027831 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.027807    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf82758-35f5-4134-b0cf-5b92a9ed951e-kube-api-access-d4wnf" pod "2cf82758-35f5-4134-b0cf-5b92a9ed951e" (UID: "2cf82758-35f5-4134-b0cf-5b92a9ed951e"). InnerVolumeSpecName "kube-api-access-d4wnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:43:19.028068 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.028048    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-xks-ca" pod "2cf82758-35f5-4134-b0cf-5b92a9ed951e" (UID: "2cf82758-35f5-4134-b0cf-5b92a9ed951e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:43:19.127243 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.127154    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4wnf\" (UniqueName: \"kubernetes.io/projected/2cf82758-35f5-4134-b0cf-5b92a9ed951e-kube-api-access-d4wnf\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:19.127243 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.127187    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf82758-35f5-4134-b0cf-5b92a9ed951e-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:19.821007 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.820946    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4" event={"ID":"2cf82758-35f5-4134-b0cf-5b92a9ed951e","Type":"ContainerDied","Data":"4de19a0a71d65a98614a93afb32ab76da211a1a20050a27558a8dba71d259ccb"}
Nov 03 20:43:19.821238 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.821021    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4de19a0a71d65a98614a93afb32ab76da211a1a20050a27558a8dba71d259ccb"
Nov 03 20:43:19.821238 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:19.820962    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"
Nov 03 20:43:20.447455 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:20.447398    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"]
Nov 03 20:43:26.803793 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:26.803706    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"]
Nov 03 20:43:26.810072 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:26.809122    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b0085-7r2j4"]
Nov 03 20:43:27.615935 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:27.615897    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf82758-35f5-4134-b0cf-5b92a9ed951e" path="/var/lib/kubelet/pods/2cf82758-35f5-4134-b0cf-5b92a9ed951e/volumes"
Nov 03 20:43:28.796223 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:43:28.796187    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:43:35.471355 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.471296    5600 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp" podUID="f6c7d443-695f-4d43-9293-33160e77157f" containerName="main" containerID="cri-o://33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437" gracePeriod=45
Nov 03 20:43:35.623426 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.623388    5600 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp" podUID="f6c7d443-695f-4d43-9293-33160e77157f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": dial tcp 10.134.0.180:8000: connect: connection refused"
Nov 03 20:43:35.723271 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.723207    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:43:35.792719 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.792682    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-tmp-dir\") pod \"f6c7d443-695f-4d43-9293-33160e77157f\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") "
Nov 03 20:43:35.792921 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.792767    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-model-cache\") pod \"f6c7d443-695f-4d43-9293-33160e77157f\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") "
Nov 03 20:43:35.792921 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.792838    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c7d443-695f-4d43-9293-33160e77157f-kube-api-access-w4v5h\" (UniqueName: \"kubernetes.io/projected/f6c7d443-695f-4d43-9293-33160e77157f-kube-api-access-w4v5h\") pod \"f6c7d443-695f-4d43-9293-33160e77157f\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") "
Nov 03 20:43:35.792921 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.792906    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6c7d443-695f-4d43-9293-33160e77157f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6c7d443-695f-4d43-9293-33160e77157f-tls-certs\") pod \"f6c7d443-695f-4d43-9293-33160e77157f\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") "
Nov 03 20:43:35.793108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.792961    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-home\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-home\") pod \"f6c7d443-695f-4d43-9293-33160e77157f\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") "
Nov 03 20:43:35.793108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.792991    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-dshm\") pod \"f6c7d443-695f-4d43-9293-33160e77157f\" (UID: \"f6c7d443-695f-4d43-9293-33160e77157f\") "
Nov 03 20:43:35.793108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.792973    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-tmp-dir" pod "f6c7d443-695f-4d43-9293-33160e77157f" (UID: "f6c7d443-695f-4d43-9293-33160e77157f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:43:35.793108 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.793046    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-model-cache" pod "f6c7d443-695f-4d43-9293-33160e77157f" (UID: "f6c7d443-695f-4d43-9293-33160e77157f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:43:35.793311 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.793275    5600 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-tmp-dir\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:35.793311 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.793294    5600 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-model-cache\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:35.793446 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.793320    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-home" pod "f6c7d443-695f-4d43-9293-33160e77157f" (UID: "f6c7d443-695f-4d43-9293-33160e77157f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:43:35.795316 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.795292    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-dshm" pod "f6c7d443-695f-4d43-9293-33160e77157f" (UID: "f6c7d443-695f-4d43-9293-33160e77157f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:43:35.795529 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.795471    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c7d443-695f-4d43-9293-33160e77157f-kube-api-access-w4v5h" pod "f6c7d443-695f-4d43-9293-33160e77157f" (UID: "f6c7d443-695f-4d43-9293-33160e77157f"). InnerVolumeSpecName "kube-api-access-w4v5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:43:35.795529 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.795473    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c7d443-695f-4d43-9293-33160e77157f-tls-certs" pod "f6c7d443-695f-4d43-9293-33160e77157f" (UID: "f6c7d443-695f-4d43-9293-33160e77157f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:43:35.894630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.894599    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4v5h\" (UniqueName: \"kubernetes.io/projected/f6c7d443-695f-4d43-9293-33160e77157f-kube-api-access-w4v5h\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:35.894630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.894627    5600 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6c7d443-695f-4d43-9293-33160e77157f-tls-certs\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:35.894630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.894636    5600 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-home\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:35.894848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.894645    5600 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6c7d443-695f-4d43-9293-33160e77157f-dshm\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:35.914178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.914150    5600 generic.go:356] "Generic (PLEG): container finished" podID="f6c7d443-695f-4d43-9293-33160e77157f" containerID="33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437" exitCode=0
Nov 03 20:43:35.914338 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.914192    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp" event={"ID":"f6c7d443-695f-4d43-9293-33160e77157f","Type":"ContainerDied","Data":"33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437"}
Nov 03 20:43:35.914338 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.914201    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"
Nov 03 20:43:35.914338 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.914218    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp" event={"ID":"f6c7d443-695f-4d43-9293-33160e77157f","Type":"ContainerDied","Data":"25046d96ad908ec87168cf5ea5b57383555b91b806b42c9b9c7f142829aad8ad"}
Nov 03 20:43:35.914338 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.914239    5600 scope.go:122] "RemoveContainer" containerID="33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437"
Nov 03 20:43:35.924570 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.924554    5600 scope.go:122] "RemoveContainer" containerID="33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437"
Nov 03 20:43:35.924839 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:43:35.924816    5600 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437\": container with ID starting with 33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437 not found: ID does not exist" containerID="33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437"
Nov 03 20:43:35.924907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.924847    5600 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437"} err="failed to get container status \"33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437\": rpc error: code = NotFound desc = could not find container \"33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437\": container with ID starting with 33d771cfea09d32565df02b6e6680eb6f840327130b366efb2e828d2d4e9c437 not found: ID does not exist"
Nov 03 20:43:35.936092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.936068    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"]
Nov 03 20:43:35.939867 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:35.939842    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-4b0085/simulator-single-main-4b0085-kserve-6f778dbc8c-w2vvp"]
Nov 03 20:43:37.615734 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:43:37.615696    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c7d443-695f-4d43-9293-33160e77157f" path="/var/lib/kubelet/pods/f6c7d443-695f-4d43-9293-33160e77157f/volumes"
Nov 03 20:43:58.875712 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:43:58.875664    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:44:08.019677 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:44:08.019639    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"]
Nov 03 20:44:08.021867 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:44:08.021838    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2d6539-bhw79"]
Nov 03 20:44:09.615120 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:44:09.615086    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc53c6e-96d6-4907-b241-a6d89a05abf7" path="/var/lib/kubelet/pods/cbc53c6e-96d6-4907-b241-a6d89a05abf7/volumes"
Nov 03 20:44:28.842460 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:44:28.841638    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:44:58.796754 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:44:58.796669    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:45:22.364934 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:22.364899    5600 scope.go:122] "RemoveContainer" containerID="abadb878f53f76f64bbe920059e9d417c003f9558d09703def8683d1213fce55"
Nov 03 20:45:22.373948 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:22.373924    5600 scope.go:122] "RemoveContainer" containerID="b7db9e3ae04baaaf91e45b3701bcaf191fe3b2bc24f7efd547b06ed167c7e659"
Nov 03 20:45:26.953261 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.953218    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"]
Nov 03 20:45:26.953709 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.953689    5600 cpu_manager.go:414] "containerMap: removing container" podUID="2cf82758-35f5-4134-b0cf-5b92a9ed951e" containerName="revoke-keys"
Nov 03 20:45:26.953709 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.953703    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf82758-35f5-4134-b0cf-5b92a9ed951e" containerName="revoke-keys"
Nov 03 20:45:26.953793 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.953719    5600 cpu_manager.go:414] "containerMap: removing container" podUID="f6c7d443-695f-4d43-9293-33160e77157f" containerName="main"
Nov 03 20:45:26.953793 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.953727    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c7d443-695f-4d43-9293-33160e77157f" containerName="main"
Nov 03 20:45:26.953858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.953804    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf82758-35f5-4134-b0cf-5b92a9ed951e" containerName="revoke-keys"
Nov 03 20:45:26.953858 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.953816    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c7d443-695f-4d43-9293-33160e77157f" containerName="main"
Nov 03 20:45:26.958490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.958471    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:26.966639 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.966616    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:45:26.966821 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.966711    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:45:26.969460 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.969402    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"]
Nov 03 20:45:26.995307 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.995267    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsz7t\" (UniqueName: \"kubernetes.io/projected/2abddc6c-9e41-49d4-aae3-0daf2544ae93-kube-api-access-wsz7t\") pod \"maas-api-revoke-keys-e2e-ait-364642e1-gt6vl\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:26.995523 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.995390    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-364642e1-gt6vl\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:26.995575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:26.995546    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-364642e1-gt6vl\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:27.096701 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:27.096660    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsz7t\" (UniqueName: \"kubernetes.io/projected/2abddc6c-9e41-49d4-aae3-0daf2544ae93-kube-api-access-wsz7t\") pod \"maas-api-revoke-keys-e2e-ait-364642e1-gt6vl\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:27.096893 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:27.096742    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-364642e1-gt6vl\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:27.096893 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:27.096823    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-364642e1-gt6vl\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:27.097473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:27.097417    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-364642e1-gt6vl\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:27.099365 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:27.099343    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-364642e1-gt6vl\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:27.102681 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:45:27.102659    5600 projected.go:196] Error preparing data for projected volume kube-api-access-wsz7t for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 20:45:27.102811 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:45:27.102739    5600 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2abddc6c-9e41-49d4-aae3-0daf2544ae93-kube-api-access-wsz7t podName:2abddc6c-9e41-49d4-aae3-0daf2544ae93 nodeName:}" failed. No retries permitted until 2026-11-03 20:45:27.602721015 +0000 UTC m=+3910.115542796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wsz7t" (UniqueName: "kubernetes.io/projected/2abddc6c-9e41-49d4-aae3-0daf2544ae93-kube-api-access-wsz7t") pod "maas-api-revoke-keys-e2e-ait-364642e1-gt6vl" (UID: "2abddc6c-9e41-49d4-aae3-0daf2544ae93") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 20:45:27.603282 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:27.603242    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsz7t\" (UniqueName: \"kubernetes.io/projected/2abddc6c-9e41-49d4-aae3-0daf2544ae93-kube-api-access-wsz7t\") pod \"maas-api-revoke-keys-e2e-ait-364642e1-gt6vl\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:27.614490 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:27.614464    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsz7t\" (UniqueName: \"kubernetes.io/projected/2abddc6c-9e41-49d4-aae3-0daf2544ae93-kube-api-access-wsz7t\") pod \"maas-api-revoke-keys-e2e-ait-364642e1-gt6vl\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:27.873580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:27.873488    5600 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl" secret="" err="secret \"maas-api-cleanup-dockercfg-zlwvw\" not found"
Nov 03 20:45:27.873580 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:27.873572    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:28.147153 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:28.147124    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"]
Nov 03 20:45:28.149702 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:45:28.149673    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abddc6c_9e41_49d4_aae3_0daf2544ae93.slice/crio-6a3358c8843d9563f0722ba3ba8d143f130ecaec2983d6aa5c8f3aac9b8e64e4 WatchSource:0}: Error finding container 6a3358c8843d9563f0722ba3ba8d143f130ecaec2983d6aa5c8f3aac9b8e64e4: Status 404 returned error can't find the container with id 6a3358c8843d9563f0722ba3ba8d143f130ecaec2983d6aa5c8f3aac9b8e64e4
Nov 03 20:45:28.740084 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:28.739993    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl" event={"ID":"2abddc6c-9e41-49d4-aae3-0daf2544ae93","Type":"ContainerStarted","Data":"b112ca37ec352cb9f3f3a4546981be454a8e16590b6f8a3602669e0f5e28320f"}
Nov 03 20:45:28.740084 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:28.740050    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl" event={"ID":"2abddc6c-9e41-49d4-aae3-0daf2544ae93","Type":"ContainerStarted","Data":"6a3358c8843d9563f0722ba3ba8d143f130ecaec2983d6aa5c8f3aac9b8e64e4"}
Nov 03 20:45:28.740289 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:28.740264    5600 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl" secret="" err="secret \"maas-api-cleanup-dockercfg-zlwvw\" not found"
Nov 03 20:45:28.756831 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:28.756774    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl" podStartSLOduration=2.756758709 podStartE2EDuration="2.756758709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:28.756032171 +0000 UTC m=+3911.268853953" watchObservedRunningTime="2026-11-03 20:45:28.756758709 +0000 UTC m=+3911.269580509"
Nov 03 20:45:28.795390 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:45:28.795350    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:45:29.747175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:29.747129    5600 generic.go:356] "Generic (PLEG): container finished" podID="2abddc6c-9e41-49d4-aae3-0daf2544ae93" containerID="b112ca37ec352cb9f3f3a4546981be454a8e16590b6f8a3602669e0f5e28320f" exitCode=0
Nov 03 20:45:29.747676 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:29.747194    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl" event={"ID":"2abddc6c-9e41-49d4-aae3-0daf2544ae93","Type":"ContainerDied","Data":"b112ca37ec352cb9f3f3a4546981be454a8e16590b6f8a3602669e0f5e28320f"}
Nov 03 20:45:30.887516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:30.887490    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:30.934415 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:30.934374    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-openshift-ca\") pod \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") "
Nov 03 20:45:30.934636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:30.934451    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abddc6c-9e41-49d4-aae3-0daf2544ae93-kube-api-access-wsz7t\" (UniqueName: \"kubernetes.io/projected/2abddc6c-9e41-49d4-aae3-0daf2544ae93-kube-api-access-wsz7t\") pod \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") "
Nov 03 20:45:30.934636 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:30.934512    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-xks-ca\") pod \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\" (UID: \"2abddc6c-9e41-49d4-aae3-0daf2544ae93\") "
Nov 03 20:45:30.934806 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:30.934777    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-openshift-ca" pod "2abddc6c-9e41-49d4-aae3-0daf2544ae93" (UID: "2abddc6c-9e41-49d4-aae3-0daf2544ae93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:30.934913 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:30.934895    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:30.936641 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:30.936613    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-xks-ca" pod "2abddc6c-9e41-49d4-aae3-0daf2544ae93" (UID: "2abddc6c-9e41-49d4-aae3-0daf2544ae93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:30.936752 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:30.936697    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abddc6c-9e41-49d4-aae3-0daf2544ae93-kube-api-access-wsz7t" pod "2abddc6c-9e41-49d4-aae3-0daf2544ae93" (UID: "2abddc6c-9e41-49d4-aae3-0daf2544ae93"). InnerVolumeSpecName "kube-api-access-wsz7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:31.035958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:31.035871    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2abddc6c-9e41-49d4-aae3-0daf2544ae93-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:31.035958 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:31.035901    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsz7t\" (UniqueName: \"kubernetes.io/projected/2abddc6c-9e41-49d4-aae3-0daf2544ae93-kube-api-access-wsz7t\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:31.759760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:31.759716    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl" event={"ID":"2abddc6c-9e41-49d4-aae3-0daf2544ae93","Type":"ContainerDied","Data":"6a3358c8843d9563f0722ba3ba8d143f130ecaec2983d6aa5c8f3aac9b8e64e4"}
Nov 03 20:45:31.759760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:31.759739    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"
Nov 03 20:45:31.759760 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:31.759767    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a3358c8843d9563f0722ba3ba8d143f130ecaec2983d6aa5c8f3aac9b8e64e4"
Nov 03 20:45:38.234472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:38.234410    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"]
Nov 03 20:45:38.236715 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:38.236687    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-364642e1-gt6vl"]
Nov 03 20:45:39.617372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:45:39.617328    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2abddc6c-9e41-49d4-aae3-0daf2544ae93" path="/var/lib/kubelet/pods/2abddc6c-9e41-49d4-aae3-0daf2544ae93/volumes"
Nov 03 20:45:58.794582 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:45:58.794540    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:46:28.796323 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:46:28.796228    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:46:58.794815 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:46:58.794772    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:47:22.929365 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:22.929322    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"]
Nov 03 20:47:22.929910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:22.929716    5600 cpu_manager.go:414] "containerMap: removing container" podUID="2abddc6c-9e41-49d4-aae3-0daf2544ae93" containerName="revoke-keys"
Nov 03 20:47:22.929910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:22.929727    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abddc6c-9e41-49d4-aae3-0daf2544ae93" containerName="revoke-keys"
Nov 03 20:47:22.929910 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:22.929813    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abddc6c-9e41-49d4-aae3-0daf2544ae93" containerName="revoke-keys"
Nov 03 20:47:22.932813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:22.932796    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:22.939521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:22.939495    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:47:22.939724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:22.939704    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6b4bp\""
Nov 03 20:47:22.942372 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:22.942347    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"]
Nov 03 20:47:23.110661 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:23.110624    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:23.110661 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:23.110660    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:23.110883 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:23.110706    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxr5\" (UniqueName: \"kubernetes.io/projected/a06d50bb-612d-4d09-89b0-a2987cf899cf-kube-api-access-ptxr5\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:23.212167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:23.212083    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:23.212167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:23.212122    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:23.212167 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:23.212166    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxr5\" (UniqueName: \"kubernetes.io/projected/a06d50bb-612d-4d09-89b0-a2987cf899cf-kube-api-access-ptxr5\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:23.212908 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:23.212888    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:23.214727 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:23.214703    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:23.219871 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:23.219837    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxr5\" (UniqueName: \"kubernetes.io/projected/a06d50bb-612d-4d09-89b0-a2987cf899cf-kube-api-access-ptxr5\") pod \"maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:23.247530 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:23.247501    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:23.522546 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:23.522518    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"]
Nov 03 20:47:23.525395 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:47:23.525361    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06d50bb_612d_4d09_89b0_a2987cf899cf.slice/crio-2d5106aefc14c5938709288fc15229cee0020f9067b50dbbb8750a43ceb82278 WatchSource:0}: Error finding container 2d5106aefc14c5938709288fc15229cee0020f9067b50dbbb8750a43ceb82278: Status 404 returned error can't find the container with id 2d5106aefc14c5938709288fc15229cee0020f9067b50dbbb8750a43ceb82278
Nov 03 20:47:24.360889 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:24.360856    5600 generic.go:356] "Generic (PLEG): container finished" podID="a06d50bb-612d-4d09-89b0-a2987cf899cf" containerID="002df3f05b7a0a27a96ca4899e56de970a3bfebd958986664cb3b6cead7ca94c" exitCode=0
Nov 03 20:47:24.361354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:24.360935    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7" event={"ID":"a06d50bb-612d-4d09-89b0-a2987cf899cf","Type":"ContainerDied","Data":"002df3f05b7a0a27a96ca4899e56de970a3bfebd958986664cb3b6cead7ca94c"}
Nov 03 20:47:24.361354 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:24.360992    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7" event={"ID":"a06d50bb-612d-4d09-89b0-a2987cf899cf","Type":"ContainerStarted","Data":"2d5106aefc14c5938709288fc15229cee0020f9067b50dbbb8750a43ceb82278"}
Nov 03 20:47:25.500721 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:25.500698    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:25.636022 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:25.635996    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06d50bb-612d-4d09-89b0-a2987cf899cf-kube-api-access-ptxr5\" (UniqueName: \"kubernetes.io/projected/a06d50bb-612d-4d09-89b0-a2987cf899cf-kube-api-access-ptxr5\") pod \"a06d50bb-612d-4d09-89b0-a2987cf899cf\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") "
Nov 03 20:47:25.636190 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:25.636070    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-openshift-ca\") pod \"a06d50bb-612d-4d09-89b0-a2987cf899cf\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") "
Nov 03 20:47:25.636190 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:25.636094    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-xks-ca\") pod \"a06d50bb-612d-4d09-89b0-a2987cf899cf\" (UID: \"a06d50bb-612d-4d09-89b0-a2987cf899cf\") "
Nov 03 20:47:25.636492 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:25.636467    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-openshift-ca" pod "a06d50bb-612d-4d09-89b0-a2987cf899cf" (UID: "a06d50bb-612d-4d09-89b0-a2987cf899cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:25.638458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:25.638416    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-xks-ca" pod "a06d50bb-612d-4d09-89b0-a2987cf899cf" (UID: "a06d50bb-612d-4d09-89b0-a2987cf899cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:25.638575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:25.638493    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06d50bb-612d-4d09-89b0-a2987cf899cf-kube-api-access-ptxr5" pod "a06d50bb-612d-4d09-89b0-a2987cf899cf" (UID: "a06d50bb-612d-4d09-89b0-a2987cf899cf"). InnerVolumeSpecName "kube-api-access-ptxr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:25.737089 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:25.737050    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:25.737089 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:25.737079    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptxr5\" (UniqueName: \"kubernetes.io/projected/a06d50bb-612d-4d09-89b0-a2987cf899cf-kube-api-access-ptxr5\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:25.737089 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:25.737092    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a06d50bb-612d-4d09-89b0-a2987cf899cf-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:26.372442 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:26.372407    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"
Nov 03 20:47:26.372602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:26.372465    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7" event={"ID":"a06d50bb-612d-4d09-89b0-a2987cf899cf","Type":"ContainerDied","Data":"2d5106aefc14c5938709288fc15229cee0020f9067b50dbbb8750a43ceb82278"}
Nov 03 20:47:26.372602 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:26.372507    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d5106aefc14c5938709288fc15229cee0020f9067b50dbbb8750a43ceb82278"
Nov 03 20:47:28.794788 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:47:28.794750    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:47:38.404222 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:38.404182    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"]
Nov 03 20:47:38.407287 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:38.407262    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-ca53c0-dhgj7"]
Nov 03 20:47:39.615246 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:39.615208    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06d50bb-612d-4d09-89b0-a2987cf899cf" path="/var/lib/kubelet/pods/a06d50bb-612d-4d09-89b0-a2987cf899cf/volumes"
Nov 03 20:47:56.881210 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:56.881124    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"]
Nov 03 20:47:56.881696 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:56.881532    5600 cpu_manager.go:414] "containerMap: removing container" podUID="a06d50bb-612d-4d09-89b0-a2987cf899cf" containerName="revoke-keys"
Nov 03 20:47:56.881696 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:56.881544    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06d50bb-612d-4d09-89b0-a2987cf899cf" containerName="revoke-keys"
Nov 03 20:47:56.881696 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:56.881600    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06d50bb-612d-4d09-89b0-a2987cf899cf" containerName="revoke-keys"
Nov 03 20:47:56.884850 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:56.884832    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:56.890150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:56.890127    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:47:56.890373 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:56.890286    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6b4bp\""
Nov 03 20:47:56.892957 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:56.892759    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"]
Nov 03 20:47:56.981088 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:56.981058    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-9pv9q\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:56.981258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:56.981104    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4st5\" (UniqueName: \"kubernetes.io/projected/4255516c-13b8-4a07-9bcc-0761a9ed73d9-kube-api-access-g4st5\") pod \"maas-api-revoke-sub-f03629db-9pv9q\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:56.981258 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:56.981175    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-9pv9q\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:57.081855 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:57.081821    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-9pv9q\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:57.082024 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:57.081870    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4st5\" (UniqueName: \"kubernetes.io/projected/4255516c-13b8-4a07-9bcc-0761a9ed73d9-kube-api-access-g4st5\") pod \"maas-api-revoke-sub-f03629db-9pv9q\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:57.082024 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:57.081893    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-9pv9q\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:57.082594 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:57.082575    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-9pv9q\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:57.084402 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:57.084380    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-9pv9q\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:57.090053 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:57.090032    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4st5\" (UniqueName: \"kubernetes.io/projected/4255516c-13b8-4a07-9bcc-0761a9ed73d9-kube-api-access-g4st5\") pod \"maas-api-revoke-sub-f03629db-9pv9q\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:57.200094 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:57.200064    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:57.471575 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:57.471547    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"]
Nov 03 20:47:57.474314 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:47:57.474284    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4255516c_13b8_4a07_9bcc_0761a9ed73d9.slice/crio-7d864838ac00fbd450cdf524a761eac7389167f582ada57b7698d88cd3a3c599 WatchSource:0}: Error finding container 7d864838ac00fbd450cdf524a761eac7389167f582ada57b7698d88cd3a3c599: Status 404 returned error can't find the container with id 7d864838ac00fbd450cdf524a761eac7389167f582ada57b7698d88cd3a3c599
Nov 03 20:47:57.583192 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:57.583153    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q" event={"ID":"4255516c-13b8-4a07-9bcc-0761a9ed73d9","Type":"ContainerStarted","Data":"7d864838ac00fbd450cdf524a761eac7389167f582ada57b7698d88cd3a3c599"}
Nov 03 20:47:58.590447 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:58.590395    5600 generic.go:356] "Generic (PLEG): container finished" podID="4255516c-13b8-4a07-9bcc-0761a9ed73d9" containerID="f2585177283eee5de6249c831cb85ec58d3eb2ec265252a29dcfa4e815cf796a" exitCode=0
Nov 03 20:47:58.590844 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:58.590476    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q" event={"ID":"4255516c-13b8-4a07-9bcc-0761a9ed73d9","Type":"ContainerDied","Data":"f2585177283eee5de6249c831cb85ec58d3eb2ec265252a29dcfa4e815cf796a"}
Nov 03 20:47:58.800909 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:47:58.794560    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:47:59.730983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:59.730953    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:47:59.810990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:59.810950    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-xks-ca\") pod \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") "
Nov 03 20:47:59.810990 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:59.810991    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4255516c-13b8-4a07-9bcc-0761a9ed73d9-kube-api-access-g4st5\" (UniqueName: \"kubernetes.io/projected/4255516c-13b8-4a07-9bcc-0761a9ed73d9-kube-api-access-g4st5\") pod \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") "
Nov 03 20:47:59.811273 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:59.811020    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-openshift-ca\") pod \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\" (UID: \"4255516c-13b8-4a07-9bcc-0761a9ed73d9\") "
Nov 03 20:47:59.811410 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:59.811387    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-openshift-ca" pod "4255516c-13b8-4a07-9bcc-0761a9ed73d9" (UID: "4255516c-13b8-4a07-9bcc-0761a9ed73d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:59.813315 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:59.813292    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4255516c-13b8-4a07-9bcc-0761a9ed73d9-kube-api-access-g4st5" pod "4255516c-13b8-4a07-9bcc-0761a9ed73d9" (UID: "4255516c-13b8-4a07-9bcc-0761a9ed73d9"). InnerVolumeSpecName "kube-api-access-g4st5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:59.813385 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:59.813295    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-xks-ca" pod "4255516c-13b8-4a07-9bcc-0761a9ed73d9" (UID: "4255516c-13b8-4a07-9bcc-0761a9ed73d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:59.912543 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:59.912507    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:59.912543 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:59.912538    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4st5\" (UniqueName: \"kubernetes.io/projected/4255516c-13b8-4a07-9bcc-0761a9ed73d9-kube-api-access-g4st5\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:59.912757 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:47:59.912553    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4255516c-13b8-4a07-9bcc-0761a9ed73d9-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:00.602130 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:00.602090    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q" event={"ID":"4255516c-13b8-4a07-9bcc-0761a9ed73d9","Type":"ContainerDied","Data":"7d864838ac00fbd450cdf524a761eac7389167f582ada57b7698d88cd3a3c599"}
Nov 03 20:48:00.602130 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:00.602113    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9pv9q"
Nov 03 20:48:00.602378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:00.602145    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d864838ac00fbd450cdf524a761eac7389167f582ada57b7698d88cd3a3c599"
Nov 03 20:48:08.020124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:08.020079    5600 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"]
Nov 03 20:48:08.023206 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:08.023165    5600 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bed252c-dd6zt"]
Nov 03 20:48:09.614868 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:09.614831    5600 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d63a767-842d-4ebd-8e26-b613e5dc33ba" path="/var/lib/kubelet/pods/9d63a767-842d-4ebd-8e26-b613e5dc33ba/volumes"
Nov 03 20:48:12.761596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.761560    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"]
Nov 03 20:48:12.761992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.761954    5600 cpu_manager.go:414] "containerMap: removing container" podUID="4255516c-13b8-4a07-9bcc-0761a9ed73d9" containerName="revoke-keys"
Nov 03 20:48:12.761992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.761965    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4255516c-13b8-4a07-9bcc-0761a9ed73d9" containerName="revoke-keys"
Nov 03 20:48:12.762073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.762032    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="4255516c-13b8-4a07-9bcc-0761a9ed73d9" containerName="revoke-keys"
Nov 03 20:48:12.765693 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.765675    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:12.777672 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.777649    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:48:12.777933 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.777917    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6b4bp\""
Nov 03 20:48:12.780583 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.780562    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"]
Nov 03 20:48:12.864309 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.864278    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bs4tj\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:12.864521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.864339    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j664f\" (UniqueName: \"kubernetes.io/projected/441544d9-989d-4fc8-9d9c-d325847e4835-kube-api-access-j664f\") pod \"maas-api-revoke-sub-0165d7d2-bs4tj\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:12.864521 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.864494    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bs4tj\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:12.965297 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.965261    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bs4tj\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:12.965556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.965355    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j664f\" (UniqueName: \"kubernetes.io/projected/441544d9-989d-4fc8-9d9c-d325847e4835-kube-api-access-j664f\") pod \"maas-api-revoke-sub-0165d7d2-bs4tj\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:12.965556 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.965463    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bs4tj\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:12.966061 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.966042    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-bs4tj\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:12.967887 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.967864    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-bs4tj\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:12.974907 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:12.974883    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j664f\" (UniqueName: \"kubernetes.io/projected/441544d9-989d-4fc8-9d9c-d325847e4835-kube-api-access-j664f\") pod \"maas-api-revoke-sub-0165d7d2-bs4tj\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:13.078196 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:13.078105    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:13.350325 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:13.350299    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"]
Nov 03 20:48:13.352541 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:48:13.352514    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441544d9_989d_4fc8_9d9c_d325847e4835.slice/crio-7d7e4c3eb9677cdfbd93b7f8b05edcbe393b2b25d5e04ed07c893c6b32f382bb WatchSource:0}: Error finding container 7d7e4c3eb9677cdfbd93b7f8b05edcbe393b2b25d5e04ed07c893c6b32f382bb: Status 404 returned error can't find the container with id 7d7e4c3eb9677cdfbd93b7f8b05edcbe393b2b25d5e04ed07c893c6b32f382bb
Nov 03 20:48:13.685980 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:13.685947    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj" event={"ID":"441544d9-989d-4fc8-9d9c-d325847e4835","Type":"ContainerStarted","Data":"ef36bb1e8d2277f14e3bcde28c76ac5ba08400bef55aa1cb8ba8c5cb32c5ae23"}
Nov 03 20:48:13.686163 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:13.686005    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj" event={"ID":"441544d9-989d-4fc8-9d9c-d325847e4835","Type":"ContainerStarted","Data":"7d7e4c3eb9677cdfbd93b7f8b05edcbe393b2b25d5e04ed07c893c6b32f382bb"}
Nov 03 20:48:13.702092 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:13.702047    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj" podStartSLOduration=1.7020335370000002 podStartE2EDuration="1.702033537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:13.701228892 +0000 UTC m=+4076.214050672" watchObservedRunningTime="2026-11-03 20:48:13.702033537 +0000 UTC m=+4076.214855317"
Nov 03 20:48:14.692472 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:14.692419    5600 generic.go:356] "Generic (PLEG): container finished" podID="441544d9-989d-4fc8-9d9c-d325847e4835" containerID="ef36bb1e8d2277f14e3bcde28c76ac5ba08400bef55aa1cb8ba8c5cb32c5ae23" exitCode=0
Nov 03 20:48:14.692872 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:14.692506    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj" event={"ID":"441544d9-989d-4fc8-9d9c-d325847e4835","Type":"ContainerDied","Data":"ef36bb1e8d2277f14e3bcde28c76ac5ba08400bef55aa1cb8ba8c5cb32c5ae23"}
Nov 03 20:48:15.842124 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:15.842097    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:15.993253 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:15.993168    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-xks-ca\") pod \"441544d9-989d-4fc8-9d9c-d325847e4835\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") "
Nov 03 20:48:15.993427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:15.993266    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-openshift-ca\") pod \"441544d9-989d-4fc8-9d9c-d325847e4835\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") "
Nov 03 20:48:15.993427 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:15.993300    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441544d9-989d-4fc8-9d9c-d325847e4835-kube-api-access-j664f\" (UniqueName: \"kubernetes.io/projected/441544d9-989d-4fc8-9d9c-d325847e4835-kube-api-access-j664f\") pod \"441544d9-989d-4fc8-9d9c-d325847e4835\" (UID: \"441544d9-989d-4fc8-9d9c-d325847e4835\") "
Nov 03 20:48:15.993694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:15.993672    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-openshift-ca" pod "441544d9-989d-4fc8-9d9c-d325847e4835" (UID: "441544d9-989d-4fc8-9d9c-d325847e4835"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:15.995564 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:15.995543    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-xks-ca" pod "441544d9-989d-4fc8-9d9c-d325847e4835" (UID: "441544d9-989d-4fc8-9d9c-d325847e4835"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:15.995673 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:15.995653    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441544d9-989d-4fc8-9d9c-d325847e4835-kube-api-access-j664f" pod "441544d9-989d-4fc8-9d9c-d325847e4835" (UID: "441544d9-989d-4fc8-9d9c-d325847e4835"). InnerVolumeSpecName "kube-api-access-j664f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:16.094301 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:16.094261    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:16.094301 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:16.094296    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/441544d9-989d-4fc8-9d9c-d325847e4835-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:16.094542 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:16.094310    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j664f\" (UniqueName: \"kubernetes.io/projected/441544d9-989d-4fc8-9d9c-d325847e4835-kube-api-access-j664f\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:16.704880 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:16.704846    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj"
Nov 03 20:48:16.705062 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:16.704896    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-bs4tj" event={"ID":"441544d9-989d-4fc8-9d9c-d325847e4835","Type":"ContainerDied","Data":"7d7e4c3eb9677cdfbd93b7f8b05edcbe393b2b25d5e04ed07c893c6b32f382bb"}
Nov 03 20:48:16.705062 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:16.704942    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d7e4c3eb9677cdfbd93b7f8b05edcbe393b2b25d5e04ed07c893c6b32f382bb"
Nov 03 20:48:23.525633 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.525599    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"]
Nov 03 20:48:23.526240 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.526072    5600 cpu_manager.go:414] "containerMap: removing container" podUID="441544d9-989d-4fc8-9d9c-d325847e4835" containerName="revoke-keys"
Nov 03 20:48:23.526240 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.526085    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441544d9-989d-4fc8-9d9c-d325847e4835" containerName="revoke-keys"
Nov 03 20:48:23.526240 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.526173    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="441544d9-989d-4fc8-9d9c-d325847e4835" containerName="revoke-keys"
Nov 03 20:48:23.529458 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.529422    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:23.533175 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.533149    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:48:23.534291 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.534272    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6b4bp\""
Nov 03 20:48:23.540983 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.540948    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"]
Nov 03 20:48:23.560286 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.560260    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-bkhd9\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:23.560470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.560302    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-bkhd9\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:23.560470 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.560321    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slqdc\" (UniqueName: \"kubernetes.io/projected/9587bf22-5093-495b-be3d-0959ce45d091-kube-api-access-slqdc\") pod \"maas-api-revoke-sub-6b19338f-bkhd9\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:23.660762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.660716    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-bkhd9\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:23.660762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.660759    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slqdc\" (UniqueName: \"kubernetes.io/projected/9587bf22-5093-495b-be3d-0959ce45d091-kube-api-access-slqdc\") pod \"maas-api-revoke-sub-6b19338f-bkhd9\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:23.661015 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.660892    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-bkhd9\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:23.661558 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.661539    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-bkhd9\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:23.663374 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.663356    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-bkhd9\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:23.670178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.670144    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slqdc\" (UniqueName: \"kubernetes.io/projected/9587bf22-5093-495b-be3d-0959ce45d091-kube-api-access-slqdc\") pod \"maas-api-revoke-sub-6b19338f-bkhd9\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:23.846884 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:23.846799    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:24.123788 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:24.123759    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"]
Nov 03 20:48:24.126136 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:48:24.126108    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9587bf22_5093_495b_be3d_0959ce45d091.slice/crio-bed1bba11c84064957f80617f28bb54109cb267f6807810cb464a5bda953c492 WatchSource:0}: Error finding container bed1bba11c84064957f80617f28bb54109cb267f6807810cb464a5bda953c492: Status 404 returned error can't find the container with id bed1bba11c84064957f80617f28bb54109cb267f6807810cb464a5bda953c492
Nov 03 20:48:24.747825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:24.747791    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9" event={"ID":"9587bf22-5093-495b-be3d-0959ce45d091","Type":"ContainerStarted","Data":"554f609d82b41ec28cfdd0a63af77f62be8598f213c65f43fe552bf932d46552"}
Nov 03 20:48:24.748362 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:24.747840    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9" event={"ID":"9587bf22-5093-495b-be3d-0959ce45d091","Type":"ContainerStarted","Data":"bed1bba11c84064957f80617f28bb54109cb267f6807810cb464a5bda953c492"}
Nov 03 20:48:24.767157 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:24.767108    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9" podStartSLOduration=1.767096258 podStartE2EDuration="1.767096258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:24.765306525 +0000 UTC m=+4087.278128308" watchObservedRunningTime="2026-11-03 20:48:24.767096258 +0000 UTC m=+4087.279918041"
Nov 03 20:48:25.755402 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:25.755364    5600 generic.go:356] "Generic (PLEG): container finished" podID="9587bf22-5093-495b-be3d-0959ce45d091" containerID="554f609d82b41ec28cfdd0a63af77f62be8598f213c65f43fe552bf932d46552" exitCode=0
Nov 03 20:48:25.755822 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:25.755471    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9" event={"ID":"9587bf22-5093-495b-be3d-0959ce45d091","Type":"ContainerDied","Data":"554f609d82b41ec28cfdd0a63af77f62be8598f213c65f43fe552bf932d46552"}
Nov 03 20:48:26.886707 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:26.886676    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:26.987619 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:26.987590    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-xks-ca\") pod \"9587bf22-5093-495b-be3d-0959ce45d091\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") "
Nov 03 20:48:26.987776 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:26.987686    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9587bf22-5093-495b-be3d-0959ce45d091-kube-api-access-slqdc\" (UniqueName: \"kubernetes.io/projected/9587bf22-5093-495b-be3d-0959ce45d091-kube-api-access-slqdc\") pod \"9587bf22-5093-495b-be3d-0959ce45d091\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") "
Nov 03 20:48:26.989825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:26.989793    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-xks-ca" pod "9587bf22-5093-495b-be3d-0959ce45d091" (UID: "9587bf22-5093-495b-be3d-0959ce45d091"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:26.989953 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:26.989869    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9587bf22-5093-495b-be3d-0959ce45d091-kube-api-access-slqdc" pod "9587bf22-5093-495b-be3d-0959ce45d091" (UID: "9587bf22-5093-495b-be3d-0959ce45d091"). InnerVolumeSpecName "kube-api-access-slqdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:27.089150 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:27.089066    5600 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-openshift-ca\") pod \"9587bf22-5093-495b-be3d-0959ce45d091\" (UID: \"9587bf22-5093-495b-be3d-0959ce45d091\") "
Nov 03 20:48:27.089321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:27.089293    5600 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slqdc\" (UniqueName: \"kubernetes.io/projected/9587bf22-5093-495b-be3d-0959ce45d091-kube-api-access-slqdc\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:27.089321 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:27.089313    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-xks-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:27.089516 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:27.089489    5600 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-openshift-ca" pod "9587bf22-5093-495b-be3d-0959ce45d091" (UID: "9587bf22-5093-495b-be3d-0959ce45d091"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:27.190378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:27.190349    5600 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9587bf22-5093-495b-be3d-0959ce45d091-maas-api-openshift-ca\") on node \"ip-10-0-128-24.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:27.768226 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:27.768190    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9" event={"ID":"9587bf22-5093-495b-be3d-0959ce45d091","Type":"ContainerDied","Data":"bed1bba11c84064957f80617f28bb54109cb267f6807810cb464a5bda953c492"}
Nov 03 20:48:27.768395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:27.768238    5600 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bed1bba11c84064957f80617f28bb54109cb267f6807810cb464a5bda953c492"
Nov 03 20:48:27.768395 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:27.768202    5600 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-bkhd9"
Nov 03 20:48:28.794477 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:48:28.794419    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:48:45.386184 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:45.386150    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-g487l_7121959d-7966-4626-addd-371fa2295cae/manager/0.log"
Nov 03 20:48:45.589254 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:45.589218    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rlfbz_ce3fa57a-f470-4d07-9cfc-1a3ece3df591/manager/0.log"
Nov 03 20:48:46.230718 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:46.230681    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zt4t7_eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8/manager/0.log"
Nov 03 20:48:46.441650 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:46.441624    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2qb6d_05f7ff11-6561-4688-b7e8-5b518643fe5a/proxy/0.log"
Nov 03 20:48:47.557895 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:47.557865    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-bs4tj_441544d9-989d-4fc8-9d9c-d325847e4835/revoke-keys/0.log"
Nov 03 20:48:48.083953 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:48.083924    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-bkhd9_9587bf22-5093-495b-be3d-0959ce45d091/revoke-keys/0.log"
Nov 03 20:48:48.447378 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:48.447342    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-9pv9q_4255516c-13b8-4a07-9bcc-0761a9ed73d9/revoke-keys/0.log"
Nov 03 20:48:48.624095 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:48.624067    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-q9b6f_51d053ff-ea16-4ffd-ac26-ad845dce0b82/postgres/0.log"
Nov 03 20:48:50.334736 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:50.334710    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-2sjs8_60ca69b0-824e-4a96-a238-7d5416576c66/manager/0.log"
Nov 03 20:48:50.521693 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:50.521665    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bk89d_de1f93a5-a1d4-4a1d-945c-f08aaa2b932a/manager/0.log"
Nov 03 20:48:50.891091 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:50.891061    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5dvbz_53f8ba01-60ee-4a82-880e-2fcf8ba98d68/manager/0.log"
Nov 03 20:48:52.510724 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:52.510694    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd88dbbb7-48gc7_20362046-2f57-4bea-8b2d-4a9092f60059/kube-auth-proxy/0.log"
Nov 03 20:48:53.227145 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:53.227082    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-4f6wl_e41ec2f8-05e2-456f-be60-a013db04b022/istio-proxy/0.log"
Nov 03 20:48:53.954720 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:53.954686    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-k6fhj_770a59d3-9646-4f7a-a152-3eeaae4cbfbf/istio-proxy/1.log"
Nov 03 20:48:54.470050 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:54.470022    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-wp8sz_c52f10e7-447a-4942-ba2e-e42509bf6e33/istio-proxy/1.log"
Nov 03 20:48:56.108989 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:56.108960    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-k6fhj_770a59d3-9646-4f7a-a152-3eeaae4cbfbf/istio-proxy/0.log"
Nov 03 20:48:56.466138 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:48:56.466099    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-wp8sz_c52f10e7-447a-4942-ba2e-e42509bf6e33/istio-proxy/0.log"
Nov 03 20:48:58.794045 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:48:58.794010    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:49:08.516110 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:08.516077    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-g487l_7121959d-7966-4626-addd-371fa2295cae/manager/0.log"
Nov 03 20:49:08.724121 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:08.724090    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rlfbz_ce3fa57a-f470-4d07-9cfc-1a3ece3df591/manager/0.log"
Nov 03 20:49:09.345596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:09.345555    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zt4t7_eb4ab8ab-5b3b-4e1f-89e0-60fbaad953a8/manager/0.log"
Nov 03 20:49:09.553452 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:09.553399    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-2qb6d_05f7ff11-6561-4688-b7e8-5b518643fe5a/proxy/0.log"
Nov 03 20:49:10.456159 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:10.456124    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-bs4tj_441544d9-989d-4fc8-9d9c-d325847e4835/revoke-keys/0.log"
Nov 03 20:49:10.971264 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:10.971229    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-bkhd9_9587bf22-5093-495b-be3d-0959ce45d091/revoke-keys/0.log"
Nov 03 20:49:11.320940 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:11.320868    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-9pv9q_4255516c-13b8-4a07-9bcc-0761a9ed73d9/revoke-keys/0.log"
Nov 03 20:49:11.493047 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:11.493011    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-q9b6f_51d053ff-ea16-4ffd-ac26-ad845dce0b82/postgres/0.log"
Nov 03 20:49:12.813998 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:12.813970    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-2sjs8_60ca69b0-824e-4a96-a238-7d5416576c66/manager/0.log"
Nov 03 20:49:12.998305 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:12.998270    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bk89d_de1f93a5-a1d4-4a1d-945c-f08aaa2b932a/manager/0.log"
Nov 03 20:49:13.337828 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:13.337799    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5dvbz_53f8ba01-60ee-4a82-880e-2fcf8ba98d68/manager/0.log"
Nov 03 20:49:14.624630 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:14.624598    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd88dbbb7-48gc7_20362046-2f57-4bea-8b2d-4a9092f60059/kube-auth-proxy/0.log"
Nov 03 20:49:15.333762 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:15.333732    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-4f6wl_e41ec2f8-05e2-456f-be60-a013db04b022/istio-proxy/0.log"
Nov 03 20:49:16.039367 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:16.039333    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-k6fhj_770a59d3-9646-4f7a-a152-3eeaae4cbfbf/istio-proxy/1.log"
Nov 03 20:49:16.581052 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:16.581023    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-wp8sz_c52f10e7-447a-4942-ba2e-e42509bf6e33/istio-proxy/1.log"
Nov 03 20:49:22.423528 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:22.423492    5600 scope.go:122] "RemoveContainer" containerID="3f8829499f463bd0a387f90d8ece1e73972dcca0be72fe8e2e9d80fa43bdcfcd"
Nov 03 20:49:22.432551 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:49:22.432532    5600 scope.go:122] "RemoveContainer" containerID="0f2aa6a65ec458b47163180acb00f4ef849bf925263cf466fa5393153c07ed70"
Nov 03 20:49:28.794555 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:49:28.794464    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:49:58.793873 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:49:58.793829    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:50:28.794331 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:50:28.794291    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:50:47.415042 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:47.415008    5600 ???:1] "http: TLS handshake error from 10.0.128.24:33516: EOF"
Nov 03 20:50:47.424764 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:47.424740    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9sbm7_e2aab3d5-357d-470a-a27d-bab9d570a73d/global-pull-secret-syncer/0.log"
Nov 03 20:50:47.729813 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:47.729727    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x8dsr_ab539183-8f67-4c14-aef9-98b5924b6c33/konnectivity-agent/0.log"
Nov 03 20:50:47.757962 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:47.757938    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-24.us-east-2.compute.internal_92c0a5455fd7c0c77f98e0c6b3176e58/haproxy/0.log"
Nov 03 20:50:54.741008 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:54.740961    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-2sjs8_60ca69b0-824e-4a96-a238-7d5416576c66/manager/0.log"
Nov 03 20:50:54.792125 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:54.792081    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bk89d_de1f93a5-a1d4-4a1d-945c-f08aaa2b932a/manager/0.log"
Nov 03 20:50:54.917825 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:54.917780    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5dvbz_53f8ba01-60ee-4a82-880e-2fcf8ba98d68/manager/0.log"
Nov 03 20:50:56.258929 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.258895    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4a52ed9-95d8-4b79-b0f2-8c230139391b/alertmanager/0.log"
Nov 03 20:50:56.281999 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.281972    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4a52ed9-95d8-4b79-b0f2-8c230139391b/config-reloader/0.log"
Nov 03 20:50:56.303773 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.303751    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4a52ed9-95d8-4b79-b0f2-8c230139391b/kube-rbac-proxy-web/0.log"
Nov 03 20:50:56.327798 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.327771    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4a52ed9-95d8-4b79-b0f2-8c230139391b/kube-rbac-proxy/0.log"
Nov 03 20:50:56.349944 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.349921    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4a52ed9-95d8-4b79-b0f2-8c230139391b/kube-rbac-proxy-metric/0.log"
Nov 03 20:50:56.372021 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.371964    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4a52ed9-95d8-4b79-b0f2-8c230139391b/prom-label-proxy/0.log"
Nov 03 20:50:56.400482 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.400453    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4a52ed9-95d8-4b79-b0f2-8c230139391b/init-config-reloader/0.log"
Nov 03 20:50:56.477991 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.477964    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rdgs7_edcd5516-336e-415d-b587-c887774d8a63/kube-state-metrics/0.log"
Nov 03 20:50:56.498078 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.498043    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rdgs7_edcd5516-336e-415d-b587-c887774d8a63/kube-rbac-proxy-main/0.log"
Nov 03 20:50:56.519694 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.519628    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rdgs7_edcd5516-336e-415d-b587-c887774d8a63/kube-rbac-proxy-self/0.log"
Nov 03 20:50:56.547073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.547050    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-cd7f87d47-wq4k7_c31a30a7-dcde-418e-8297-f5af07990cb0/metrics-server/0.log"
Nov 03 20:50:56.741729 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.741696    5600 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"]
Nov 03 20:50:56.742116 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.742103    5600 cpu_manager.go:414] "containerMap: removing container" podUID="9587bf22-5093-495b-be3d-0959ce45d091" containerName="revoke-keys"
Nov 03 20:50:56.742197 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.742116    5600 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9587bf22-5093-495b-be3d-0959ce45d091" containerName="revoke-keys"
Nov 03 20:50:56.742197 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.742194    5600 memory_manager.go:365] "RemoveStaleState removing state" podUID="9587bf22-5093-495b-be3d-0959ce45d091" containerName="revoke-keys"
Nov 03 20:50:56.745302 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.745287    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.751487 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.751464    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5pd2z\"/\"kube-root-ca.crt\""
Nov 03 20:50:56.752506 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.752479    5600 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5pd2z\"/\"openshift-service-ca.crt\""
Nov 03 20:50:56.752692 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.752673    5600 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5pd2z\"/\"default-dockercfg-pvhgd\""
Nov 03 20:50:56.754773 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.754749    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"]
Nov 03 20:50:56.785735 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.785671    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-sys\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.785735 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.785717    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-lib-modules\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.785902 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.785798    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-podres\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.785902 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.785830    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85gjk\" (UniqueName: \"kubernetes.io/projected/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-kube-api-access-85gjk\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.785902 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.785854    5600 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-proc\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.799595 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.799564    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zftbb_5d178b87-95d6-4985-bf0f-f0448dd784d7/node-exporter/0.log"
Nov 03 20:50:56.824304 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.824281    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zftbb_5d178b87-95d6-4985-bf0f-f0448dd784d7/kube-rbac-proxy/0.log"
Nov 03 20:50:56.847848 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.847827    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zftbb_5d178b87-95d6-4985-bf0f-f0448dd784d7/init-textfile/0.log"
Nov 03 20:50:56.875691 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.875656    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qpx5d_de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/kube-rbac-proxy-main/0.log"
Nov 03 20:50:56.887264 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.887231    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-sys\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.887374 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.887276    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-lib-modules\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.887374 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.887300    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-podres\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.887374 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.887316    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85gjk\" (UniqueName: \"kubernetes.io/projected/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-kube-api-access-85gjk\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.887539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.887371    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-sys\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.887539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.887392    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-podres\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.887539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.887393    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-lib-modules\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.887539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.887422    5600 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-proc\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.887539 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.887486    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-proc\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.895227 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.895195    5600 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85gjk\" (UniqueName: \"kubernetes.io/projected/79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2-kube-api-access-85gjk\") pod \"perf-node-gather-daemonset-5d9m9\" (UID: \"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:56.896992 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.896971    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qpx5d_de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/kube-rbac-proxy-self/0.log"
Nov 03 20:50:56.918745 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.918718    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qpx5d_de3eb2db-f7a4-4fd0-8fdd-e84c11d822b1/openshift-state-metrics/0.log"
Nov 03 20:50:56.952199 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.952177    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df71796f-bae7-44f2-b35a-dad7a3f03497/prometheus/0.log"
Nov 03 20:50:56.973073 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:56.973056    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df71796f-bae7-44f2-b35a-dad7a3f03497/config-reloader/0.log"
Nov 03 20:50:57.001020 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:57.000998    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df71796f-bae7-44f2-b35a-dad7a3f03497/thanos-sidecar/0.log"
Nov 03 20:50:57.024772 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:57.024745    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df71796f-bae7-44f2-b35a-dad7a3f03497/kube-rbac-proxy-web/0.log"
Nov 03 20:50:57.046154 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:57.046090    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df71796f-bae7-44f2-b35a-dad7a3f03497/kube-rbac-proxy/0.log"
Nov 03 20:50:57.059915 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:57.059896    5600 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:57.073130 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:57.073114    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df71796f-bae7-44f2-b35a-dad7a3f03497/kube-rbac-proxy-thanos/0.log"
Nov 03 20:50:57.094942 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:57.094904    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_df71796f-bae7-44f2-b35a-dad7a3f03497/init-config-reloader/0.log"
Nov 03 20:50:57.334388 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:57.334362    5600 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"]
Nov 03 20:50:57.336453 ip-10-0-128-24 kubenswrapper[5600]: W1103 20:50:57.336400    5600 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod79a3f8b8_8cb3_4ce0_bddb_342d76bd0df2.slice/crio-7427b81cfb6eb856c9db298e02e0982e467ac942324a5405ae750900fcb5b4ec WatchSource:0}: Error finding container 7427b81cfb6eb856c9db298e02e0982e467ac942324a5405ae750900fcb5b4ec: Status 404 returned error can't find the container with id 7427b81cfb6eb856c9db298e02e0982e467ac942324a5405ae750900fcb5b4ec
Nov 03 20:50:57.595596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:57.595496    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9" event={"ID":"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2","Type":"ContainerStarted","Data":"7629add59acab5d15a4e621e55dee5abfec639c82a3fa87fbd29138a145b8fb0"}
Nov 03 20:50:57.595596 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:57.595569    5600 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9" event={"ID":"79a3f8b8-8cb3-4ce0-bddb-342d76bd0df2","Type":"ContainerStarted","Data":"7427b81cfb6eb856c9db298e02e0982e467ac942324a5405ae750900fcb5b4ec"}
Nov 03 20:50:57.595845 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:57.595615    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:50:57.612641 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:50:57.612578    5600 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9" podStartSLOduration=1.612559888 podStartE2EDuration="1.612559888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:57.610935285 +0000 UTC m=+4240.123757260" watchObservedRunningTime="2026-11-03 20:50:57.612559888 +0000 UTC m=+4240.125381669"
Nov 03 20:50:58.795182 ip-10-0-128-24 kubenswrapper[5600]: E1103 20:50:58.795147    5600 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btkpj" containerName="extractor"
Nov 03 20:51:01.649076 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:01.649045    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgwk6_8668d2b7-8bf0-4786-b96f-0ba9ab45472c/dns/0.log"
Nov 03 20:51:01.669577 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:01.669555    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgwk6_8668d2b7-8bf0-4786-b96f-0ba9ab45472c/kube-rbac-proxy/0.log"
Nov 03 20:51:01.752328 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:01.752299    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zbfhf_16525d8b-c267-41c1-8048-cec8f8dc381f/dns-node-resolver/0.log"
Nov 03 20:51:02.454088 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:02.454056    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hrqf7_0034d15b-8b67-4e49-8abd-b834b164af4f/node-ca/0.log"
Nov 03 20:51:03.615079 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:03.615051    5600 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-5d9m9"
Nov 03 20:51:05.751574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:05.751531    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd88dbbb7-48gc7_20362046-2f57-4bea-8b2d-4a9092f60059/kube-auth-proxy/0.log"
Nov 03 20:51:05.880601 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:05.880562    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-4f6wl_e41ec2f8-05e2-456f-be60-a013db04b022/istio-proxy/0.log"
Nov 03 20:51:05.998260 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:05.998231    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-k6fhj_770a59d3-9646-4f7a-a152-3eeaae4cbfbf/istio-proxy/0.log"
Nov 03 20:51:05.998493 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:05.998267    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-k6fhj_770a59d3-9646-4f7a-a152-3eeaae4cbfbf/istio-proxy/1.log"
Nov 03 20:51:06.081494 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:06.081360    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-wp8sz_c52f10e7-447a-4942-ba2e-e42509bf6e33/istio-proxy/1.log"
Nov 03 20:51:06.081727 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:06.081594    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-wp8sz_c52f10e7-447a-4942-ba2e-e42509bf6e33/istio-proxy/0.log"
Nov 03 20:51:08.670387 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:08.670358    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62g8g_2224bfc1-cc82-4e90-b723-e629d5f0f3f5/serve-healthcheck-canary/0.log"
Nov 03 20:51:09.299975 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:09.299934    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-btkpj_8bedff4e-9e2b-4582-bafe-29360cfa83ae/kube-rbac-proxy/0.log"
Nov 03 20:51:09.322473 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:09.322421    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-btkpj_8bedff4e-9e2b-4582-bafe-29360cfa83ae/exporter/0.log"
Nov 03 20:51:09.345642 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:09.345616    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-btkpj_8bedff4e-9e2b-4582-bafe-29360cfa83ae/extractor/0.log"
Nov 03 20:51:09.568859 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:09.568772    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd88dbbb7-48gc7_20362046-2f57-4bea-8b2d-4a9092f60059/kube-auth-proxy/0.log"
Nov 03 20:51:09.639178 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:09.639141    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-4f6wl_e41ec2f8-05e2-456f-be60-a013db04b022/istio-proxy/0.log"
Nov 03 20:51:09.714491 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:09.714411    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-k6fhj_770a59d3-9646-4f7a-a152-3eeaae4cbfbf/istio-proxy/1.log"
Nov 03 20:51:09.714491 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:09.714490    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-k6fhj_770a59d3-9646-4f7a-a152-3eeaae4cbfbf/istio-proxy/0.log"
Nov 03 20:51:09.762346 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:09.762319    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-wp8sz_c52f10e7-447a-4942-ba2e-e42509bf6e33/istio-proxy/1.log"
Nov 03 20:51:09.762574 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:09.762489    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-wp8sz_c52f10e7-447a-4942-ba2e-e42509bf6e33/istio-proxy/0.log"
Nov 03 20:51:12.078567 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:12.078536    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5749d45fbb-lbhkn_64b489b3-2516-451a-9c2d-95914c2bed5c/manager/0.log"
Nov 03 20:51:12.099749 ip-10-0-128-24 kubenswrapper[5600]: I1103 20:51:12.099726    5600 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5749d45fbb-lbhkn_64b489b3-2516-451a-9c2d-95914c2bed5c/copy-manifests/0.log"